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EXECUTIVE  SUMMARY 


INTRODUCTION 

During  the  last  3  years,  the  Environmental  Security  Technology  Certification 
Program  (ESTCP)  has  sponsored  the  development  of  an  advanced  vehicular  unexploded 
ordnance  (UXO)  detection  system  at  the  Naval  Research  Laboratory  (NRL)  called  the 
Multi-sensor  Towed  Array  Detection  System  (MTADS).  The  MTADS  can  be  configured 
with  one  of  three  sensor  arrays:  magnetometer,  magnetic  gradiometer,  and  electro¬ 
magnetic  induction  (EMI). 

In  December  1996,  on  the  Magnetic  Test  Range  (MTR)  at  the  Marine  Corps  Air 
Ground  Combat  Center  (MCAGCC),  Twentynine  Palms,  California,  the  MTADS  under¬ 
went  its  first  semi-blind  field  test.  During  the  test,  all  three  sensors  were  used.  The 
MTADS  was  driven  over  a  test  site  in  which  ordnance  had  been  emplaced.  After 
completing  the  data  acquisition  and  determining  the  location  of  potential  ordnance,  IDA 
performed  an  assessment  to  determine  overall  detection  performance  and  unique 
characteristics  of  the  MTADS.  We  assessed  both  the  results  of  the  IDA  target-picking 
analyses  and  the  results  of  a  separate  analysis  performed  by  NRL.  This  analysis  provides 
a  comprehensive  look  at  the  overall  capabilities  of  the  MTADS  as  a  tool  for  UXO 
detection  and  location. 

This  paper  gives  the  results  of  IDA’s  and  NRL’s  analysis  and  provides  a 
performance  analysis  of  each  sensor  array. 

RESULTS 
Detection  Rate 

In  general,  the  detection  rate  for  the  magnetometer  and  magnetic  gradiometer 
systems  was  very  good  (see  Table  ES-1  and  Figure  ES-1),  considering  the  difficult 
magnetic  background  present  at  the  MTR.  The  robustness  of  the  MTADS-Data 
Acquisition  System  (DAS)  improved  these  systems’  performance.  For  the  smaller,  near¬ 
surface  mortars,  however,  the  magnetometer  and  magnetic  gradiometer  exhibited  reduced 
detection  performance.  Also,  the  magnetometer  did  not  detect  many  of  the  large,  deeply 
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Table  ES-1.  Detection  and  False-Alarm  Performance  of  MTADS 
from  IDA  and  NRL  Analysis  with  a  1-m  Rent 


IDA 

NRL 

Sensor 

P, 

FAR 

(m-2) 

Pfa 

P, 

FAR 

(m-^) 

Pf. 

Magnetometer 
(Full  Data  Set) 

0.726 

2.0  X  10'^ 

6.5  X  10'^ 

0.823 

7.4  X  10-" 

2.4x  10-" 

Magnetometer 
(Confidence  Level  2) 

0.677 

7.4  X  10'® 

2.4x10-" 

0.790 

4.9x  10-" 

1.5x  10-" 

Magnetic  Gradiometer 
(Full  Data  Set) 

0.710 

8.3x  10-® 

2.6x  10-" 

0.774 

7.7x10-" 

2.4x  10-" 

Magnetic  Gradiometer 
(Confidence  Level  2) 

0.532 

2.6  X  10-^ 

8.1  X  10-" 

0.726 

4.9x10-" 

1.6x  10-" 

EMI 

0.823 

9.5x  10“® 

2.0x  10-" 

0.839 

4.8  X  10-" 

1.5x10-" 

Figure  ES-1.  versus  P^for  NRL  Processing  (Triangles)  and  IDA  Processing  (Squares). 
The  open  marks  are  for  the  full  declaration  set.  The  solid  symbols  result  from 
implementing  the  appropriate  discrimination  algorithms  for  each  data  set  to  a 
confidence  level  of  2.  (a)  Magnetometer,  (b)  Magnetic  Gradiometer.  (c)  EMi. 

buried  bombs.  These  items  probably  were  missed  because  their  spatially  broad  signatures 
were  difficult  to  analyze  using  the  MTADS-DhS,  given  the  magnetic  geological  clutter  at 
the  MTR. 
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The  detection  rate  for  the  EMI  was  better  than  that  of  the  magnetometer  or  the 
magnetic  gradiometer.  The  EMI  system  detected  all  the  smaller  near-surface  ordnance, 
missing  only  the  deepest  ordnance.  The  EMI  achieved  this  level  of  performance  because 
of  good  location  accuracy,  a  result  of  using  the  real-time  kinematic-differential  global 
positioning  system  (RTK-DGPS)  as  a  navigation  system. 

Location  Accuracy 

All  sensor  modalities  exhibited  excellent  location  accuracy  (see  Table  ES-2). 
Both  the  magnetometer  and  the  EMI  had  a  mean  location  accuracy  better  than  30  cm. 
Analysis  of  the  magnetometer  and  magnetic  gradiometer  data  showed  excellent  depth 
estimations.  The  EMI  system  (including  the  EMI  analysis  portion  of  the  MTADS-DAS), 
however,  systematically  underestimated  the  ordnance  depth. 


Table  ES-2.  Detection  Location  Error  and  Depth  Error  for  1-m  Critical  Radius'^ 


Radial 

Depth 

Magnitude  Depth 

Sensor 

Mean 

Standard 

Mean 

Standard 

Mean 

Standard 

Error 

Deviation 

Error 

Deviation 

Error 

Deviation 

(m) 

(m) 

(m) 

(m) 

(m) 

(m) 

Magnetometer 

0.28 

0.24 

0.10 

0.39 

0.26 

0.31 

Magnetic 

Gradiometer 

0.37 

0.26 

0.09 

0.66 

0.28 

0.61 

EMI 

0.27 

0.16 

-0.19 

0.52 

0.37 

0.41 

Data  is  determined  from  the  anomalies  chosen  by  NRL. 


False-Alarm  Rate 

We  used  the  relative  inclination  of  the  magnetic  dipole  moment  of  the  anomalies 
(calculated  using  the  MTADS-DAS)  as  a  discriminant  on  the  magnetic  data,  resulting  in 
greater  than  30-percent  reduction  (for  the  IDA  targets  picked)  in  the  number  of  false 
alarms  for  both  the  magnetometer  (see  Figure  ES-2)  and  the  magnetic  gradiometer  data. 
This  technique  also  resulted  in  a  small  reduction  of  the  number  of  detections.  Because  the 
quality  of  the  fit  magnetic  moment  data  was  poor,  the  magnetic  gradiometer  did  not 
exhibit  as  large  a  performance  increase  using  this  technique  as  the  magnetometer. 

Conclusions 

Overall,  the  MTADS  exhibited  excellent  performance  for  detection  of  buried 
UXO.  In  addition,  the  quality  of  the  data  collected  by  the  MTADS  permits  the  develop¬ 
ment  and  implementation  of  advanced  descrimination  algorithms  to  help  mitigate  the 
false-alarm  problem  that  plagues  most  detections  systems. 
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Figure  ES-2.  Distribution  of  Magnetic  Moments  Relative  to  the  Geomagnetic  Field  as 
Measured  by  the  Magnetometer.  Relative  inclination  of  90  deg  is  aligned  with  the 
geomagnetic  field.  Relative  inclination  of  0  deg  is  orthogonal  to  geomagnetic  field. 

(a)  Ordnance,  (b)  False  alarms. 
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1.0  INTRODUCTION 


During  the  last  3  years  the  Environmental  Security  Technology  Certification 
Program  (ESTCP)  has  sponsored  the  development  of  an  advanced  vehicular  unexploded 
ordnance  detection  (UXO)  system  at  the  Naval  Research  Laboratory  (NRL).  This  vehicle, 
the  Multi-sensor  Towed  Array  Detection  System  (MTADS),  underwent  its  first  semi-blind 
field  test  at  the  Marine  Corps  Air  Ground  Combat  Center  (MCAGCC),  Twentynine 
Palms,  California,  in  December  1996.  The  test  was  conducted  at  the  NRL  Magnetic  Test 
Range  (MTR)  located  within  the  confines  of  the  MCAGCC.  The  Institute  for  Defense 
Analyses  (IDA)  used  data  acquired  during  that  semi-blind  test  to  locate  the  emplaced 
ordnance  (before  release  of  the  emplaced  ordnance  baseline  information),  i.e.,  to  pick 
targets. 

After  completing  the  data  acquisition  and  determining  the  location  of  potential 
ordnance,  IDA  performed  an  assessment  to  determine  overall  detection  performance  and 
unique  characteristics  of  the  MTADS.  We  assessed  both  the  results  of  the  IDA  target¬ 
picking  analyses  and  the  results  of  a  separate  analysis  performed  by  NRL.  In  this 
document,  the  focus  is  on  the  analysis  tool,  but  since  the  ability  to  perform  good  analysis 
is  dependent  on  the  quality  of  data,  we  will  also  comment  on  the  strengths  and  weak¬ 
nesses  of  the  MTADS  platform. 

1.1  THE  MTADS 

The  MTADS  consists  of  a  low-magnetic  tow  vehicle  and  two  carts  which  carry 
either  magnetometer/magnetic  gradiometer  arrays  or  an  electromagnetic  induction  (EMI) 
array.  The  MTADS  is  designed  to  minimize  the  noise  and  signal  bias  caused  by  the  tow 
vehicle  and  the  cart,  while  providing  a  capable  and  versatile  detection  system.  Extensive 
development  has  focused  on  the  quality  of  the  test  data,  with  navigation  being  a  para¬ 
mount  issue.  The  vehicle  and  sensor  position  are  determined  using  a  real-time  kinematic 
differential  global  position  system  (RTK-DGPS).  The  RTK-DGPS  has  been  integrated 
with  great  care  to  ensure  that  all  sensor  time  stamps  are  accurate  and  synchronized  with 
the  RTK-DGPS.  The  result  is  a  location  accuracy  of  approximately  5  cm.  This  high  level 
of  location  accuracy  provides  a  unique  capability  to  co-register  data  collected  at  different 
times,  with  different  sensors  or  sensor  configurations,  over  the  same  survey  site. 
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Software  tools  have  been  developed  to  provide  a  dynamic  data  acquisition  system 
(MTADS-DAQ);  also  developed  is  a  data  analysis  tool  called  the  MTADS  data  analysis 
system  (MTADS-DAS).  The  MTADS-DAS  permits  the  user  to  visually  process  both 
magnetostatic  and  EMI  data  collected  with  the  MTADS-DAQ.  The  user  can  preprocess 
any  of  the  raw  data  to  deal  with  loss  of  good  signal  from  the  RTK-DGPS.  In  addition,  the 
MTADS-DAS  provides  a  series  of  tool  boxes  that  permit  the  user  to  hand  pick  anomalies  • 
within  the  sensor  data  and  determine  a  list  of  features  or  parameters  associated  with  that 
anomaly.  Those  parameters  provide  a  basis  for  relocation  and  remediation  of  all 
suspected  UXO.  The  parameter  set  is  dependent  on  the  sensor  being  towed.  For  the 
magnetometer  and  magnetic  gradiometer  the  parameter  set  includes  magnitude  of  the  fit 
magnetic  moment,  inclination  and  declination  of  the  fit  magnetic  moment,  depth, 
northing  coordinate,  easting  coordinate,  and  goodness  of  fit  (GoF).^  For  the  EMI  system 
the  set  includes  depth,  northing  coordinate,  easting  coordinate,  ferrous  and  nonferrous 
size,  and  GoF.  Along  with  detection  performance  (this  includes  both  detection  and  false- 
alarm  rates),  the  fitted  parameters  for  each  declaration  are  analyzed. 

The  next  step  in  developing  a  robust  UXO  detection  system  immune  to  the 
staggering  false-alarm  rates  (FAR)  that  currently  plague  the  UXO  detection  and 
remediation  process  is  to  implement  a  discrimination  tool.  This  tool  lets  the  user  separate 
potentially  hazardous  objects — the  UXO — from  a  large  set  of  the  naturally  occurring  and 
anthropic  clutter.  In  this  report,  IDA  discusses  the  piece  of  a  discrimination  tool  under 
development  that  attempts  to  reduce  false  alarms  caused  by  magnetic  clutter  (Nelson 
et  al.,  1998). 

1.2  THE  MCAGCC  FIELD  TEST 

During  the  first  2  weeks  in  December  1996,  an  NRL  team  surveyed  the  MTR  at 
the  MCAGCC.  The  MTR  contains  70  emplaced  inert  ordnance  items  ranging  from  60- 
mm  mortars  to  2,000-lb  bombs.  Emplaced  approximately  10  years  ago,  the  ordnance  has 
been  used  by  NRL  for  previous  UXO  sensor  system  development.  NRL  has  full 
knowledge  of  the  original  location  of  all  emplaced  items.  Since  this  site  is  in  an  area  that 
sees  both  dismounted  and  vehicle  activity,  there  is  the  potential  that  some  near-surface 


'  A  more  recent  version  of  the  MTADS-DAS  has  a  rudimentary  automatic  target-picking  feature. 

^  A  measure  of  size  (or  radius  of  a  sphere  of  equivalent  volume)  is  also  provided,  but  this  is  directly 

proportional  to  the  fit  magnetic  moment  magnitude. 
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ordnance  has  shifted  slightly.  There  is  no  indication  from  the  test  results  that  any 
ordnance  has  been  removed. 

The  site  exhibits  a  high  level  of  both  geological  and  anthropic  clutter.  The 
Marines  have  littered  the  site  with  a  broad  range  of  waste,  including  spent  cartridges, 
cans,  and  communication  wire.  Although  the  presence  of  this  material  complicates  the 
detection  process,  it  reflects  real-world  conditions  and  therefore  provides  a  good  test  of 
the  MTADS.  The  magnetic  geological  clutter  provides  a  more  difficult  environment  in 
which  to  detect  UXO  using  the  magnetometer,  but  it  does  not  prevent  the  use  of  the 
sensor. 


Additional  small  ordnance  and  rebar  markers  were  emplaced  during  the  test  along 
the  MTR  perimeter  to  assess  the  minimum  detection  performance  and  act  as  fiducials, 
respectively.  This  document  does  not  use  this  data  and  removes  both  the  emplaced  items 
from  the  baseline  and  any  target  declarations  associated  with  these  emplaced  items. 

1.3  TEST  DATA 

Three  sets  of  test  data  were  collected  during  the  2  weeks: 

•  Magnetic  data — taken  using  eight  total  field  magnetometers  in  a  linear  array 
with  0.25-cm  spacing  between  magnetometers  and  a  nominal  ground-to- 
sensor  height  of  0.25  m. 

•  Magnetic  gradiometric  data — taken  using  eight  total  field  magnetometers  con¬ 
figured  as  four  vertical  magnetic  gradiometers.  Each  magnetic  gradiometer 
has  a  pair  of  magnetometers  with  0.5-m  vertical  separation.  The  magnetic 
gradiometers  are  separated  linearly  by  0.5  m.  The  lower  magnetometer  in  each 
magnetic  gradiometer  has  a  nominal  distance  above  ground  of  0.25  m. 

•  EMI  data — ^taken  using  three  modified  Geonics,  Ltd.,  EM61s  nested  in  a  2-m 
linear  array. 

Once  collected,  this  data  was  preprocessed  by  a  member  of  the  NRL  team^  to 
remove  any  navigational  errors  caused  by  loss  of  RTK-DGPS.  This  preprocessed  data, 
along  with  the  MTADS-DAS,  was  then  provided  to  IDA  for  analysis.  IDA  used  the 
MTADS-DAS  to  pick  suspected  ordnance  and  determine  the  characteristics  of  each 


3  The  preprocessing  was  done  by  a  subcontractor  to  NRL.  This  subcontractor,  AETC,  Inc.,  also 
developed  and  wrote  the  MTADS-DAS. 
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anomaly.  In  addition,  NRL  also  analyzed  the  data  and  provided  IDA  a  list  of  suspected 
ordnance  items.^ 

Upon  completion  of  the  target-picking  analysis  of  the  preprocessed  data  and 
delivery  of  the  NRL  list  of  suspected  ordnance,  IDA  was  provided  a  full  list  of  emplaced 
ordnance.  Using  this  baseline  knowledge,  an  in-depth  analysis  of  the  performance  of 
MTADS  at  the  MTR  was  completed.  This  analysis  appears  later  in  this  document. 

Chapter  2  provides  a  description  of  the  picking  process  and  processing  used  by 
IDA.  Chapter  3  presents  analyses  of  the  detection  performance  for  the  list  of  target 
declarations  picked  by  IDA  for  each  of  the  three  sensor  types.  Chapter  4  gives  a  similar 
assessment  of  detection  performance  for  the  NRL  list  of  target  declarations  and  compares 
this  to  the  IDA  targets  chosen.  The  overall  assessment  is  summarized  in  Chapter  5.  In 
addition,  an  appendix  provides  a  list  of  expected  magnetic  signatures  for  each  of  the 
emplaced  ordnance  items  in  the  ground  using  the  prolate  spheroidal  model  employed  by 
IDA  (Altshuler,  1996). 


^  This  is  not  a  true  blind  test  for  NRL  because  the  same  personnel  were  responsible  for  the  original 
emplacement  of  the  ordnance.  But  it  is  apparent  that  an  accurate  attempt  was  made  to  analyze  the 
sensor  data  without  employing  the  knowledge  of  the  baseline. 
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2.0  IDA  TARGET-PICKING  AND  MAGNETIC-MOMENT 

ANALYSES 


NRL  provided  IDA  a  full  copy  of  the  MTADS-DAS  and  the  three  principal  data 
sets  collected  at  the  MTR  located  at  the  MCAGCC  at  Twentynine  Palms.  The  three  data 
sets — magnetometer,  magnetic  gradiometer,  and  EMI — had  been  preprocessed  by  AETC, 
Inc.  (a  subcontractor  and  member  of  the  MTADS  team)  to  remove  or  correct  bad  location 
information  caused  by  loss  of  RTK-DGPS  signal.  The  post-processed  data  was  then 
ported  to  a  Silicon  Graphics  workstation  for  analysis  by  IDA.  During  the  initial  target¬ 
picking  phase,  IDA  had  no  knowledge  of  the  baseline  of  targets  emplaced  at  the  MTR. 
Thus,  the  target-picking  process  was  blind. 

Since  the  version  of  the  MTADS-DAS  used  for  this  analysis  did  not  contain  an 
automatic  anomaly-picking  feature,  anomalies  were  selected  manually  by  the  analyst. 
Once  a  final  list  of  potential  targets  was  generated,  IDA  passed  the  list  to  NRL  for  assess¬ 
ment.  At  that  time,  NRL  provided  IDA  with  the  baseline  of  targets  emplaced  at  the  MTR 
and  NRL’s  own  list  of  targets  chosen  using  the  MTADS-DAS.  Using  the  baseline  pro¬ 
vided  by  NRL,  IDA  performed  an  extensive  analysis  of  the  targets  chosen  by  each 
analysis  team  from  each  of  the  data  sets.  This  analysis  appears  in  Chapters  3  and  4  of  this 
document.  This  chapter  describes  the  process  used  by  IDA  to  pick  anomalies  from  the 
data  sets  and  generate  its  list  of  potential  ordnance. 

2.1  TARGET-PICKING  PROCESS 

Each  data  set  generated  during  the  test  of  MTADS  at  the  MTR  was  processed 
independently.  For  all  data,  the  site  was  divided  using  20-m  x  20-m  grid  cells  to  simplify 
analysis.  Point  sensor  data,  rather  than  interpolated  data,  was  viewed.  To  ensure  that 
knowledge  gained  during  the  target-picking  phase  for  one  sensor  was  not  applied  during 
the  target-picking  phase  of  a  second  sensor,  the  data  sets  were  divided  into  two  groups; 
passive  magnetic  (magnetometry  and  magnetic  gradiometry)  and  EMI,  and  a  different 
analyst  was  responsible  for  picking  the  targets  from  each  group.  The  picking  processes 
used  for  the  three  data  sets  are  described  below. 
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2.1.1  Magnetometer  Data 

The  magnetometer  data  set  was  processed  first.  Using  two  different  threshold 
levels,  a  total  of  676  anomalies  were  chosen.  The  site  map  was  completely  analyzed  at 
the  first  threshold,  set  at  100  y  above  or  below  the  average  background.  Any  compact 
feature  in  the  image  of  the  measured  magnetic  field  that  exceeded  the  threshold  was 
chosen  for  fitting  using  the  MTADS-DAS  dipole-fitting  routine.  The  process  consisted  of 
defining  a  box  surrounding  the  anomaly  to  be  processed,  then  using  the  magnetic-dipole- 
fitting  routine  to  generate  a  set  of  fit  parameters  from  the  data  within  the  boxed  region.  At 
the  100-y  threshold,  357  anomalies  were  cho.sen.  Once  the  data  was  recorded,  a  lower 
threshold — 30  y  above  or  below  the  average  background — was  used,  and  319  additional 
anomalies  were  chosen.  These  thresholds  were  not  stringent  because  the  analyst  used  a 
color  map  of  the  magnetic  field  to  identify  potential  anomalies  and  segment  the  image. 
Thus,  it  is  possible  that  anomalies  with  peak  values  slightly  below  a  threshold  were 
chosen.  Also  note  that  both  positive  and  negative  threshold  exceedences  were  chosen.  If 
the  positive  and  negative  exceedences  were  spatially  close,  they  were  fit  as  a  single 
segmented  anomaly.  Other  than  using  a  very  rough  shape  filter  (to  eliminate  large 
spatially  extended  features)  and  the  threshold,  no  attempt  was  made  to  discriminate 
during  the  picking  phase. 

Once  the  anomaly  was  chosen,  a  magnetic-dipole-fitting  routine  was  used  to 
extract  the  anomaly  features: 

•  location  (northing  and  easting); 

•  depth; 

•  ferrous  size,  given  as  the  radius  of  a  solid  sphere  that  would  yield  the  same 
size  anomaly; 

•  magnitude  of  the  dipole  moment; 

•  inclination  and  declination  of  the  magnetic  dipole  moment;  and 

•  GoF  of  the  dipole  model  used  to  generate  the  other  fitting  parameters. 

The  MTADS-DAS  permits  the  user  to  eliminate  small  sections  from  the  region  chosen  for 
analysis  and  detrend  the  data  to  optimize  the  GoF.  Both  these  features  were  used  and  the 
best  GoF  chosen  as  the  final  fit.  Note  that  no  attempt  was  made  at  either  threshold  to 
adjust  the  extent  of  the  region  chosen  to  find  an  optimum  fit;  however,  at  the  lower 
threshold  (±30  y)  if  it  was  apparent  to  the  analyst  that  a  section  of  the  anomaly  was 
outside  the  fitting  region,  the  region  was  redrawn  and  refit.  The  new  fit  parameters  were 
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then  optimized  and  used  to  provide  a  second  fit  for  the  anomaly.  Both  fits  were  analyzed 
and  the  best  was  chosen  in  post-processing. 

As  discussed  in  Chapter  3,  using  a  single  fit  region  at  a  given  threshold  resulted  in 
a  number  of  missed  targets  because  of  the  large  location  errors,  which  suggests  that  the  fit 
region  should  be  adjusted  during  analysis  to  optimize  the  fit.  For  manual  target  picking, 
optimizing  the  fit  means  that  multiple  fits  should  be  recorded  and  the  final  fit  parameters 
chosen  from  this  group. 

2.1.2  Magnetic  Gradiometer  Data 

The  magnetic  gradiometer  data  was  processed  in  the  same  manner  as  the  magne¬ 
tometer  data,  except  that  three  thresholds  were  used  (±50  y,  +30  y,  and  ±15  y).  Using 
these  three  thresholds,  a  total  of  302  anomalies  were  identified,  with  205  chosen  at  ±50  y, 
75  chosen  at  ±30  y,  and  22  chosen  at  ±15  y.  This  list  of  anomalies  was  used  to  analyze  the 
performance  of  the  magnetic  gradiometer. 

2.1.3  EMI  Data 

The  first  criterion  for  ranking  targets  was  magnitude  of  return,  so  it  was  decided 
that  the  initial  target  list  should  include  only  the  most  intense  returns.  The  false-color 
scale  and  threshold  were  varied  several  times  to  find  a  combination  expected  to  produce 
approximately  200  target  nominations.  Random  grid  cells  were  sampled  at  each  step  to 
estimate  the  total  number  of  targets  that  would  be  selected.  From  this  procedure,  the 
false-color  scale  was  set  at  5  to  100  mV.  To  be  included  in  the  target  nomination  list,  a 
return  was  required  to  exhibit  80-mV  color.  In  addition,  a  few  other  peaks  that  were 
weaker  (as  low  as  ~40  mV)  but  very  large  spatially  were  included  in  the  initial  threshold 
because  it  was  discovered  that  these  returns  generally  resulted  from  large,  deep  items. 
This  initial  screening  resulted  in  174  target  nominations.  After  the  entire  image  had  been 
processed,  the  procedure  was  repeated  using  the  5-30  mV  false-color  scale,  and  78 
additional  targets  were  added  to  the  initial  list. 

Each  anomaly  nominated  was  then  fit  using  an  EMI  fit  routine  (McDonald  et  al., 
1997).  The  procedure  for  defining  the  region  to  fit  and  the  fitting  process  are  similar  to 
those  used  for  the  magnetometer  and  magnetic  gradiometer.  The  following  fit  parameters 
were  determined  for  each  anomaly: 

•  location  (northing  and  easting); 

•  depth; 
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ferrous  and  nonferrous  size,  given  as  the  radius  of  a  solid  sphere  that  would 
yield  the  same  size  anomaly;  and 

GoF  of  the  dipole  model  used  to  generate  the  other  fitting  parameters. 


2.2  MAGNETIC  DIPOLE  MOMENT  FALSE-ALARM 
MITIGATION  TECHNIQUE 

After  the  generation  of  the  target  nominations  for  each  of  the  data  sets,  the 
magnetic  and  magnetic  gradiometer  data  were  analyzed  using  a  discrimination  algorithm 
developed  at  IDA  (Nelson  et  al.,  1998).  This  discrimination  algorithm  applies  a  con¬ 
fidence  rating  determined  by  the  inclination  of  the  fitted  magnetic  moment  relative  to  the 
geomagnetic  field.  The  final  confidence  for  each  anomaly  was  also  influenced  by  the 
GoF  and  whether  the  fitted  depth  of  the  anomaly  was  zero.  Although  both  the  depth  and 
the  GoF  are  reasonable  and  common  tools  for  discrimination,  the  use  of  the  magnetic 
moment  direction  is  novel.  An  in-depth  description  of  the  theory  underlying  the  magnetic 
moment  direction  discrimination  will  be  published  by  IDA. 


2.2.1  The  Magnetic  Signature  of  Ordnance 


Previous  work  (Altshuler,  1996)  has  demonstrated  that  the  induced  magnetization 
of  the  ordnance  item  is  dependent  on  the  size,  shape,  and  orientation  relative  to  the  geo¬ 
magnetic  field.  To  a  lesser  extent,  the  casing  thickness  also  contributes  to  the  signature. 
In  most  cases,  the  magnetic  signature  of  ordnance  is  dominated  by  the  dipole  term,  which 
is  proportional  to  where  x  is  the  distance  from  the  center  of  volume  of  the  ordnance  to 
the  magnetometer.  Higher  moments  contribute  to  the  measured  signature  when  the  stand¬ 
off  distance  of  the  sensor  is  approximately  the  length  of  the  ordnance  item.  In  the  dipole 
approximation,  the  ith  component  of  the  induced  magnetization,  M„  of  the  ordnance  item 
must  be  calculated  using  the  ith  component  of  the  magnetization  of  the  prolate  spheroid: 


M  - 

*  47t+(p-l)A^,- 


where  Hg  is  the  geomagnetic  field  strength  and  A,  is  the  ith  component  of  the  demagne¬ 
tization  factor.  In  truth,  ordnance  is  not  a  perfect  prolate  spheroid  but  is  more  generally  a 
body  of  revolution.  Still,  the  extent  of  the  asymmetry  through  the  central  plane  of  the 
ordnance  item  is  small,  and  thus  the  quadrupole  moment  resulting  from  this  asymmetry 
should  also  be  small.  So  the  assumption  that  the  signature  is  accurately  described  by  only 
the  dipole  term  is  valid. 
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Because  ordnance  casings  are  made  of  ferromagnetic  materials,  there  is  a 
potential  for  the  ordnance  to  have  a  permanent  magnetization,  called  a  remanent 
magnetization.  The  simple  model  discussed  in  the  previous  section  ignores  any  remanent 
contribution  to  the  total  magnetization.  In  reality,  the  total  magnetization,  and  thus  the 
magnetic  signature,  is  highly  dependent  on  the  magnetic  history  of  each  specific 
ordnance  item.  It  can  be  argued  that  if  the  largest  magnetic  field  the  ordnance  item  ever 
encounters  is  the  geomagnetic  field,  then  the  remanent  magnetization  is  small  compared 
to  the  induced  magnetization.  On  the  other  hand,  if  the  ordnance  item  is  subjected  to 
large  magnetic  fields  relative  to  the  geomagnetic  field,  it  is  possible  that  a  large  remanent 
signature  could  exist.  Since  the  magnetization  is  dependent  on  the  entire  magnetic 
history,  however,  even  ordnance  exposed  to  large  magnetic  fields  before  deployment 
(i.e.,  firing,  launching,  or  dropping)  will  experience  substantial  shock  upon  ground 
impact,  which  in  turn  causes  the  microscopic  magnetic  structure,  the  magnetic  domain,  to 
rearrange  to  a  lower  energy  state,  resulting  in  a  shock  demagnetization.  The  shock 
demagnetization  will  leave  the  ordnance  with  only  a  small  remanent  magnetization  and  a 
signature  dominated  by  the  induced  magnetization.  There  are  still  a  number  of  circum¬ 
stances  that  can  cause  ordnance  to  have  a  large  remanent  signature  after  ground  impact. 
These  include  lightning  strikes  near  the  ordnance. 

If  it  is  assumed  that  the  ordnance  has  impacted  the  ground  or  has  not  been 
exposed  to  large  magnetic  fields,  it  is  possible  to  estimate  the  upper  bound  for  the 
remanent  contribution  to  the  ordnance  magnetization.  This  estimate  can  be  made  by  using 
a  prolate  spheroidal  approximation  for  the  shape  of  ordnance  and  modeling  the  magneti¬ 
zation  process.  In  this  model,  it  is  assumed  that  the  ordnance  is  exposed  to  a  magnetic 
field  parallel  to  the  long  axis,  the  easiest  direction  of  magnetization,  which  results  in  the 
strongest  effective  magnetic  field  in  the  metal  casing.  Since  the  geomagnetic  field  is 
small  (approximately  0.5  Oe),  the  magnetization  process  is  very  nearly  reversible.  It  is 
the  slight  deviation  from  reversibility  at  these  applied  magnetic  field  levels  that  causes 
hysteresis  and  the  resulting  remanent  magnetization.  Back-of-the-envelope  estimations  of 
the  remanent  signature  suggest  that,  in  most  cases,  it  is  no  more  than  20  percent  of  the 
maximum  induced  magnetization.  For  this  document,  we  will  use  this  estimate  as  an 
upper  limit  for  ordnance. 

2.2.2  Expected  Distribution  of  Magnetic  Signatures 

Assuming  that  the  maximum  remanent  signature  for  ordnance  cannot  exceed 
20  percent  of  the  magnetization  induced  by  the  geomagnetic  field,  it  is  possible  to  limit 
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the  orientation  of  the  magnetic  dipole  relative  to  a  small  solid  angle.  The  size  of  the  solid 
angle  is  dependent  on  the  aspect  ratio  of  the  ordnance  item.  Table  2-1  gives  the  inclina¬ 
tion  of  the  magnetization  of  an  ordnance  item  relative  to  geomagnetic  coordinates  (see 
Figure  2-1)^  at  four  remanent  magnetizations  (0,  10,  20,  and  30  percent)  and  three  aspect 
ratios  (3,  4,  and  5).  The  relative  inclination  is  the  angle  between  the  magnetization  and  a 
plane  orthogonal  to  the  geomagnetic  field.  The  angles  are  generated  assuming  that  the 
remanent  signature  is  aligned  along  the  long  axis  of  the  ordnance  item;  its  projection  can 
be  parallel  or  antiparallel  to  the  geomagnetic  field. 


Table  2-1.  Maximum  Angle  (Degrees)  of  Magnetization  Relative  to  Geomagnetic  Field 


Aspect 

Ratio 

No  remanent 
magnetization 

10-percent 

remanent 

20-percent 

remanent 

30-percent 

remanent 

3  to  1 

54.1 

49.0 

43.8 

38.4 

4  to  1 

46.1 

40.1 

35.4 

30.0 

5  to  1 

40.6 

35.2 

29.8 

24.2 

Into  ^geo 

Ground 


Figure  2-1.  The  Relative  Inclination  Used  to  Determine  the  Direction  of 
Magnetization  in  the  Geomagnetic  Coordinate  System 


^  We  use  a  relative  coordinate  system  that  is  slightly  different  from  the  geophysical  coordinate  system. 
In  the  geophysical  coordinate  system,  the  x-axis  is  north,  the  y-axis  is  east,  and  the  z-axis  is  into  the 
ground.  Our  relative  coordinate  system  is  rotated  such  that  the  z-axis  is  aligned  with  the  geomagnetic 
field.  This  results  in  a  90-deg  relative  inclination  of  the  geomagnetic  field  (with  no  declination).  Thus, 
if  the  difference  between  the  inclination  of  the  magnetization  and  the  geomagnetic  field  is  30  deg,  then 
the  relative  inclination  of  that  magnetization  is  60  deg.  This  results  in  azimuthal  symmetry  when  doing 
our  magnetization  modeling. 
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Clutter,  on  the  other  hand,  is  not  restricted  in  angular  distribution  relative  to 
geomagnetic  coordinate.  This  assumption  is  based  on  the  substantially  different  magnetic 
histories  of  both  naturally  occurring  and  anthropic  clutter.  Most  nonordnance  materials 
have  not  been  subject  to  shock  demagnetization.  In  addition,  the  manufacturing  processes 
for  many  nonordnance  items  (rebar,  banding,  drums,  etc.)  tend  to  form  stronger  remanent 
magnetization  relative  to  that  formed  by  the  casting  process  used  to  manufacture 
ordnance.  It  is  unlikely  that  the  angular  distribution  of  clutter  signatures  will  be  uniform. 
Instead,  there  will  be  a  distribution  more  heavily  weighted  toward  the  pure  induced 
angular  distribution,  with  an  extended  tail  that  results  from  a  strong  remanent  contribu¬ 
tion  to  the  signature. 

To  estimate  the  distribution  expected  for  ordnance  and  clutter,  we  use  a  Monte 
Carlo  technique  to  estimate  the  relative  inclination  distribution  for  a  prolate  spheroid  with 
an  aspect  ratio  of  4  to  1.  We  assume  that  the  remanent  magnetization  could  be  any  value 
between  0  and  20  percent  of  the  maximum  induced  magnetization.  The  remanent 
magnetization  is  fixed  along  the  semimajor  axis,  and  the  ordnanee  direction  is  allowed  to 
take  any  direction  covering  471  steradians.  Figure  2-2(a)  shows  the  resulting  distribution. 
This  distribution  is  consistent  with  Table  2-1. 


Relative  Inclination  (degrees) 


0.02 


Relative  Inclination  (degrees) 

Figure  2-2.  Monte  Carlo  Analysis  of  Distribution  of 
Relative  Inclinations  for  Ordnance  and  Clutter 
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Figure  2-2(b)  shows  the  resulting  distribution  for  clutter;  it  was  calculated  using 
the  method  described  for  the  ordnance,  except  an  aspect  ratio  ranging  between  1  to  10 
and  10  to  1  was  assumed.  Here,  the  remanent  signature  can  take  a  value  between  0  and 
300  percent  of  the  induced  magnetization  along  the  semimajor  axis.  This  figure  shows 
that  clutter  can  exhibit  relative  inclination  covering  -90  deg  to  +90  deg.  Thus,  it  is 
possible  to  use  the  expected  relative  inclination  of  ordnance  as  a  discrimination  tech¬ 
nique.  The  relative  inclination  values  used  for  the  IDA-processed  and  NRL-processed 
data  are  described  in  Chapters  3  and  4,  respectively. 


2-8 


3.0  IDA  DECLARATION  ANALYSIS 


This  chapter  contains  an  analysis  of  target  declarations  selected  by  IDA  using  the 
MTADS-DAS.  These  declarations  are  a  result  of  analysis  of  the  three  complete  survey 
runs  over  the  MTR  at  Twentynine  Palms.  For  each  sensor  type  and  configuration 
(magnetometer,  magnetic  gradiometer,  and  EMI),  detection  (using  the  full  baseline)  and 
false-alarm  performance  is  discussed.  Since  the  MTADS-DAS  provides  more  information 
than  just  the  simple  northing/easting  position  of  the  declaration,  it  is  possible  to  evaluate 
the  characteristics  of  the  MTADS  platform  and  the  MTADS-DAS.  Along  with  detection 
performance  (this  includes  both  detection  rates  and  false-alarm  rates),  the  fitted 
parameters  for  each  declaration  are  analyzed.  In  addition,  trends  and  coimnon  features  for 
both  detections  and  false  alarms  are  presented  for  the  magnetometer  and  EMI  pair  only. 
Although  the  magnetic  gradiometer*  data  is  interesting,  few  differences  were  observed 
between  it  and  the  magnetometer  data.  Therefore,  we  feel  that  a  comparison  using  the 
magnetic  gradiometer  would  be  redundant. 

This  chapter  is  broken  down  into  four  sections.  The  first  two  sections  present  the 
detection  and  false-alarm  rates  and  a  general  analysis  of  the  IDA-developed  false-alarm 
discrimination  algorithm,  which  uses  the  fitted  magnetic  moment  direction  determined 
from  the  magnetometer  and  magnetic  gradiometer  data.  The  third  section  presents  an 
analysis  of  the  location  accuracy  relative  to  fitted  parameters  available  from  the  MTADS- 
DAS.  The  final  section  presents  an  evaluation  of  the  trends  in  both  detection  and  false- 
alarm  space  for  the  magnetometer  and  EMI  pair.  It  should  be  noted  throughout  this 
document  that  performance  of  the  three  sensor  types  is  discussed  without  explicit  dis¬ 
cussion  of  the  role  of  the  MTADS-DAS.  In  all  cases,  when  we  refer  to  the  performance  of 
a  specific  sensor,  that  assessment  includes  the  processing  feature  provided  by  the 
MTADS-DAS  and  also  any  other  additional  processing  on  the  MTADS-DAS  output. 


‘  The  magnetic  gradiometer  and  magnetometer  use  the  same  eight  total  field  magnetometers.  The 
difference  is  just  the  configuration  of  the  array. 
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3.1  GENERAL  PERFORMANCE  RESULTS 


Each  declaration  data  set  generated  by  IDA  using  the  MTADS-DAS  is  compared 
with  the  emplaced  target  baseline.  To  match  a  target  declaration  to  the  baseline,  we 
determine  if  the  relative  northing/easting  position  of  the  declaration  is  within  the  critical 
radius,  of  the  center  of  the  emplaced  ordnance  item  (in  the  north-east  plane)  (see 
Figure  3-1).  If  the  declaration  falls  outside  the  declaration  is  classified  as  a  false 
alarm. 


Figure  3-1.  Detection  of  the  Ordnance  item  is  Dependent  on  the  Relative  Location 
of  the  Declaration  (Marked  with  a  Cross)  and  the  Center  of  Volume  of  the 
Ordnance  Item.  If  the  declaration  is  within  the  critical  radius  Rcrii,  the 
ordnance  is  detected.  If  the  declaration  is  outside  Rcrit,  the  declaration 
is  a  false  alarm.  If  multiple  declarations  are  located  within  Rcrit,  only 
the  closest  counts  as  a  detection.  The  others  are  classified 
as  neither  detections  nor  false  alarms. 


3.1.1  Detection  and  False-Alarm  Rates 

A  total  of  70  ordnance  items  are  emplaced  at  the  MTR.  Of  these  items,  eight  are 
located  in  proximity  to  one  another,  that  is,  at  less  than  1.5  m  surface  distance^  (see 
Table  3-1).  An  additional  pair  of  ordnance  items  is  located  with  a  separation  of  2.6  m. 


^  Measurements  of  distances  are  broken  into  two  types:  (1)  depths  measured  from  ground  level  to  the 
center  of  volume  of  the  ordnance  item  emplace  and  (2)  surface  distances  measured  as  the  projection  of 
the  distance  in  the  north-east  plane  from  a  point  to  the  center  of  volume  of  the  ordnance  item 
emplaced.  In  this  document,  we  refer  to  the  first  as  depth  and  the  second  as  location. 
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Table  3-1.  Ordnance  Item  Groups 


Type- 
Ordnance  #1 

Type- 
Ordnance  #2 

Surface 
Distance  (m) 

Depth 

Difference  (m) 

155-mm  Projectile 

155-mm  Projectile 

1.00 

0.0 

105-mm  Projectile 

105-mm  Projectile 

0.64 

0.0 

81 -mm  Mortar 

81 -mm  Mortar 

1.30 

0.0 

155-mm  Projectile 

105-mm  Projectile 

0.0 

0.12 

250-lb  bomb 

250-lb  bomb 

2.6 

0.06 

The  ability  to  resolve  targets  is  a  function  of  depth,  signal  to  noise,  and  density  of  sensor 
sampling  over  the  targets.  The  two  individual  ordnance  items  constituting  each  of  the 
four  groups  with  small  separation  (<1.5  m)  are  either  unresol vable  by  current  sensor  tech¬ 
nology  and  processing  or,  at  best,  very  stressing  for  the  systems.  Although  the  goal  of  this 
assessment  is  to  analyze  the  performance  of  the  MTADS,  inclusion  of  the  four  groups  in 
the  baseline  has  the  potential  of  producing  statistics  on  the  depth  and  location  accuracy, 
and  the  magnetic  moment,  that  will  be  misleading.  All  performance  statistics  presented 
are  determined  using  the  62  remaining  emplaced  ordnance  items.  Any  declarations  that 
are  within  /?^„r  of  the  four  groups  will  be  discarded  from  both  the  detection  and  false- 
alarm  lists. 

The  two  250-lb  bombs  separated  by  2.6  m  present  a  more  complex  situation.  If 
they  were  shallow,  they  too  would  be  resolvable  by  all  three  sensor  types.  But  since  the 
resolvability  is  dependent  on  depth  (in  this  case,  approximately  2  m)  and  separation, 
resolution  of  these  as  individual  items  is  difficult.  Still,  we  feel  that  it  is  within  the 
capability  of  the  sensor  used  on  the  MTADS  to  resolve  these  targets.  Thus,  we  will 
include  them  as  two  separate  targets  in  the  baseline  for  this  assessment. 

It  should  be  noted  that  in  past  assessments  (for  example,  Altshuler  et  al.,  1995), 
grouped  sets  of  targets  have  been  included  in  the  baseline  as  a  single  item  (called  a 
group).  The  goal  was  to  focus  on  performance  using  the  probability  of  detection  and 
false-alarm  rate  as  metrics.  In  this  document,  we  are  evaluating  a  far  greater  set  of 
performance  parameters,  as  well  as  the  software  processing.  Thus,  the  exclusion  of  the 
grouped  targets  does  not  diminish  the  results  of  the  present  assessment. 

A  final  declaration  set  is  produced  for  each  sensor  type.  Detection  probabilities 
and  false-alarm  rates  can  then  be  determined  for  the  three  sets  of  data.  Table  3-2  lists  P^, 
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Table  3-2.  Detection  and  False-Alarm  Performance  of  MTADS 


Sensor^ 

P, 

FAR 

(m-^) 

0.5  m 

P,s 

P, 

FAR 

(m-2) 

1.0  m 

Pf. 

Magnetometer 
(Full  Data  Set) 

0.581 

2.1  X  10"^ 

1.6x10-2 

0.726 

2.0x10-2 

6.5  X  10-2 

Magnetometer 
(Confidence  Level  2) 

0.548 

7.8x  10-® 

6.1  X  10“® 

0.677 

7.4  X  10  ® 

2.4x10-2 

Magnetometer 
(Confidence  Level  1 ) 

0.532 

5.5x10'^ 

4.3  X  10“® 

0.645 

5.2  X  10-® 

1.6x  10“2 

Magnetometer 
(Confidence  Level  0) 

0.468 

2.8  X  10"^ 

2.2  X  10“® 

0.548 

2.6  X  10“® 

8.4x  10-® 

Magnetic  Gradiometer 
(Full  Data  Set) 

0.581 

8.6x10“® 

7.0x  10“® 

0.710 

8.3  X  10“® 

2.6x10-2 

Magnetic  Gradiometer 
(Confidence  Level  2) 

0.468 

2.7x  10-® 

2.1  X  10“® 

0.532 

2.6  X  10“® 

8.1  X  10“® 

Magnetic  Gradiometer 
(Confidence  Level  1) 

0.435 

2.2x10-® 

1.8x10“® 

0.468 

2.2  X  10“® 

6.8x  10“® 

Magnetic  Gradiometer 
(Confidence  Level  0) 

0.371 

1.2x10“® 

1.0  X  10“® 

0.403 

1.2  X  10® 

3.7  X  10“® 

Electromagnetic 

Induction 

0.758 

9.8  X  10“® 

3.3  X  10-® 

0.823 

9.5  X  10“® 

2.0  X  10-2 

^  The  declaration  set  generated  using  the  MTADS-DAS  is  considered  the  full  data  set.  Magnetic  moment 
modeling  and  depth  criteria  were  used  to  reduce  the  number  of  declarations.  Confidence  Level  2  refers 
to  the  set  of  declarations  with  a  final  IDA  ranking  of  2  or  less.  Confidence  Level  1  refers  to  the  set  of 
declarations  with  a  final  IDA  ranking  of  1  or  less.  Confidence  Level  0  refers  to  the  set  of  declarations 
with  a  final  IDA  ranking  of  0. 

FAR,  and  for  an  of  0.5  m  and  1.0  m.^  Here,  the  probability  of  false  alarm,  is 
represented  by  a  surrogate,  the  fraction  of  the  test  site  covered  by  false-alarm  areas  with 
radius. 4  In  the  past,  IDA  has  used  a  number  of  methods  to  calculate  surrogates  for  Pj^^ 
(Altshuler  et  al.,  1995;  Andrews  et  al.,  1996).  All  these  methods  are  based  on  the  charac¬ 
teristic  area  of  a  detection  (in  this  case  related  to  In  this  analysis,  the  FAR  is  small 


^  In  previous  performance  evaluations  of  other  UXO  detection  systems  including  the  Jefferson  Proving 
Ground  Phase  I  [Altshuler,  et  al.  (1995)],  a  critical  radius  of  2  m  was  used.  This  large  was  required 
because  of  the  navigation  errors  associated  with  the  systems  involved.  The  MTADS  uses  RTK-DGPS 
with  accuracy  of  approximately  5  cm  (McDonald  et  al.,  1997).  The  result  of  this  and  the  M7AD5-DAS 
is  that  target  location  accuracy  is  substantially  better  than  other  systems.  Therefore,  a  smaller  is 
justified  and  appropriate. 

^  In  the  strict  sense,  Pfa  should  be  determined  as  the  number  of  false  alarms  divided  by  the  opportunities 
for  a  false  alarm.  We  use  a  surrogate  for  Pfa  because  we  were  unable  to  accurately  determine  the 
opportunities  for  false  alarms  on  the  site.  We  emphasize  that  the  number  we  use  is  a  surrogate  and 
should  not  be  misinterpreted  as  the  true  Pf.r 
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relative  to  the  size  of  the  characteristic  detection  area.  Thus,  overlap  of  these  areas 
associated  with  each  false  alarm  does  not  result  in  a  significant  error.  In  the  case  of  high 
FAR,  it  is  critical  to  account  for  overlap  to  ensure  that  the  surrogate  is  bounded  by 
one. 

We  use  the  surrogate  for  to  evaluate  performance  of  the  detector  sys 
tern  utilizing  the  Receiver  Operating  Characteristic  (ROC)  approach  (Altshuler  et  al., 
1995  and  1997).  The  FAR  is  calculated  assuming  an  approximately  8-acre  test  site 
(30,686.6  m^).  No  attempt  has  been  made  to  account  for  areas  not  covered  by  MTADS. 
It  should  be  noted  that  the  FAR  decreases  as  the  critical  radius  increases  because 
declarations  originally  outside  are  now  counted  as  detections  instead  of  false  alarms. 
On  the  other  hand,  Pfa  increases  as  Pcn/is  increased.  This  is  a  penalty  resulting  from 
requiring  more  area  to  be  associated  with  each  declaration.  Since  a  doubling  of 
results  in  a  factor  of  approximately  4  increase  in  Pfa  (with  only  a  small  increase  in  P^),  it 
is  important  to  be  careful  when  comparing  at  different  P<.„,s. 

The  ROC  approach  provides  a  mechanism  to  evaluate  the  relative  performance  of 
systems  or  algorithms  for  a  given  P„„.  Figures  3-2  and  3-3  show  P^  versus  Pf^  for  three 
sensors  at  of  0.5  m  and  1.0  m,  respectively.  The  solid  lines  are  isoperformance  curves 
calculated  using  a  Gaussian  model  for  the  distribution  functions  of  the  target  and 
clutter/noise  signatures  (Van  Trees,  1968;  Altshuler  et  al.,  1997).  The  multiple  points  for 
the  magnetometer  presented  for  each  P„„  are  related  to  the  false-alarm  discrimination 
algorithm  discussed  in  Section  3.1.3.  Looking  at  an  of  0.5  m,  for  the  three  points 
representing  confidence  levels  0,  1,  and  2,  which  result  from  implementation  of  the 
discrimination  algorithm,  the  performance  does  not  improve  significantly  over  the 
different  confidence  levels  (the  three  points  fall  on  a  single  isoperformance  curve  within 
statistical  error).  But  there  is  improved  performance  using  the  discrimination  algorithm 
over  that  of  the  full  magnetometer  data  set.  The  same  trend  holds  for  P^„,  of  1.0  m. 

The  performance  of  the  magnetic  gradiometer  using  the  ROC  approach  for  P^„r  of 
0.5  m  and  1.0  m  does  not  exhibit  the  improvement  seen  for  the  magnetometer.  All  points, 
except  when  the  most  stringent  discrimination  criterion  is  applied,  fall  on  a  single 
isoperformance  curve  within  statistical  error. 

The  EMI  sensor  has  the  best  overall  performance,  exceeding  even  the  magnetom¬ 
eter  and  magnetic  gradiometer  performance  after  implementation  of  the  IDA  discrimi¬ 
nation  algorithm.  The  general  performance  trend  for  the  magnetometer  appears  slightly 
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Figure  3-2.  versus  for  the  Magnetometer  (Diamonds),  Magnetic  Gradiometer 
(Circies),  and  EMI  System  (Triangle).  R„„  is  0.5  m.  For  the  magnetometer  and 
magnetic  gradiometer,  confidence  levels  of  0, 1 ,  and  2  and  full  data  set 
are  shown  as  diamonds  and  circles,  respectively,  moving  from  left  to  right. 


Figure  3-3.  P,,  versus  Pfalor  the  Magnetometer  (Diamonds),  Magnetic  Gradiometer 
(Circles),  and  EMI  System  (Triangle).  is  1.0  m.  For  the  magnetometer  and 
magnetic  gradiometer,  confidence  levels  of  0, 1,  and  2  and  full  data  set  are 
shown  as  diamonds  and  circles,  respectively,  moving  from  left  to  right. 
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better  than  that  of  the  magnetic  gradiometer  (for  data  where  false-alarm  discrimination 
has  been  used).  But  the  statistical  significance  of  this  trend  is  hard  to  determine  because 
of  the  small  sample  size. 

It  is  possible  to  evaluate  the  performance  of  the  system  by  determining  the 
isoperformance  variable,  d,  for  each  point  and  including  a  measure  of  the  confidence 
interval  around  d.^  Figure  3-4  shows  the  isoperformance  variable  for  each  of  the  data 
point  in  Figure  3-3.  The  confidence  intervals  are  determined  assuming  that  the  detection 
process  follows  a  binomial  distribution  (Andrews  et  ah,  1996;  Simonson,  1998).  From 
Figure  3-4  it  is  apparent  that  all  three  sensor  types  perform  equally  as  well  upon  imple¬ 
mentation  of  the  IDA  magnetic  moment  discrimination  algorithm  (sometimes  referred  to 
as  a  filter),  if  one  accounts  for  statistical  uncertainty.  It  should  be  noted  that  the 
uncertainty  in  the  d  for  the  EMI  system  appears  to  be  barely  overlapping  the  magnetom¬ 
eter  and  magnetic  gradiometer.  Given  that  the  binomial  estimation  is  a  worst-case 
estimation  (Simonson,  1998),  and  given  the  experience  from  other  tests  of  EMI  and 
magnetometers  (Nelson  et  ah,  1997),  it  is  reasonable  to  assume  that  the  EMI  provides 
better  detection  capability  than  does  the  magnetometer  for  the  generic  shallow  ferrous 
target. 


3.1.2  Performance  Dependence  on  Critical  Radius 

As  discussed  above,  the  probability  of  detection  and  false-alarm  rate  are 
dependent  on  the  used  to  determine  detections.  Figures  3-5(a)-(c)  show  the  proba¬ 
bility  of  detection  and  number  of  detections  as  a  function  of  for  the  magnetometer, 
magnetic  gradiometer,  and  EMI  systems,  respectively.  It  is  apparent  that  for  the  magnetic 
detection  systems,  the  number  of  detections  does  not  increase  above  an  of  approxi¬ 
mately  0.75  m.  The  electromagnetic  detection  system  exhibits  even  better  Pj  versus 
performance,  with  no  significant  increase  in  detection  rates  above  approximately  0.5  m. 

Table  3-3  gives  the  mean  and  standard  deviation  of  the  radial  location  accuracy 
and  the  depth  accuracy^  (both  true  and  absolute)  for  all  three  sensor  configurations  for  a 
1-m  critical  radius.  The  EMI  system  exhibits  the  best  location  accuracy  but  also  shows 
the  largest  depth  error,  with  a  trend  of  underestimating  the  depth  of  the  ordnance  item.  In 


5  d  is  a  measure  of  the  separation  of  two  Gaussian  distributions  that  characterize  the  ordnance  signature 
and  the  clutter/noise,  d  is  in  units  of  standard  deviation  of  the  Gaussian  distributions. 

^  Both  accuracy  and  error  are  used  in  this  document  to  refer  to  the  difference  in  the  location  and  depth 
determined  by  the  MTADS-DAS  fitting  algorithm  and  the  baseline  emplaced  position  of  ordnance. 
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Figure  3-4.  Relative  Performance  of  Confidence  Levels  and  Configurations  at  the  MTR. 
The  performance  measure  is  in  terms  of  d,  the  separation  of  the  noise/clutter  and  full 
site  distribution  functions  assuming  a  Gaussian  behavior.  The  90-percent 
confidence  intervais  are  calculated  using  a  binomial  model  for  detection. 


Critical  Radius  (meters) 


Figure  3-5.  Detection  Performance  versus  Critical  Radius  for  the  Emplaced  Targets, 
(a)  Magnetometer,  (b)  Magnetic  Gradiometer.  (c)  EMI  System. 
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Table  3-3.  Detection  Location  Error  and  Depth  Error  for  1-m  Critical  Radius 


Sensor 

Radial 

Error 

(m) 

Radial 

Standard 

Deviation 

(m) 

Depth 

Error 

(m) 

Depth 

Standard 

Deviation 

(m) 

Magnitude 

Depth 

Error 

(m) 

Magnitude 

Depth 

Standard 

Deviation 

(m) 

Mag. 

0.30 

0.27 

0.02 

0.40 

0.23 

0.29 

Grad. 

0.32 

0.24 

0.04 

0.45 

0.24 

0.37 

EMI 

0.22 

0.16 

-0.18 

0.46 

0.33 

0.36 

all  cases  the  MTADS  platform  and  MTADS-T>KS  perform  quite  well.  The  magnetometer 
has  the  best  depth  estimation  capability,  with  a  mean  misestimation  of  approximately 
20  cm. 


3.1.3  False-Alarm  Mitigation  Techniques 

A  false-alarm  mitigation  technique  developed  by  IDA  is  used  to  sort  potential 
ordnanee  from  nonordnanee  for  both  the  magnetometer  and  magnetie  gradiometer 
declaration  set.  The  false-alarm  mitigation  uses  a  confidence-ranking  scheme  that  gives  a 
weight  to  each  declaration  based  on  the  orientation  of  the  magnetie  moment,  the  size  of 
the  magnetic  moment,  and  the  GoF.  In  addition,  declarations  determined  to  be  on  the 
surfaee  are  ranked  with  the  lowest  eonfidenee  level  beeause  the  test  did  not  include 
surfaee  ordnance,”^  but  did  include  surface  elutter.  Therefore,  even  with  some  statistical 
uncertainty  of  the  depth  estimate,  this  technique  should  provide  a  reasonable  method  to 
mitigate  the  high  false-alarm  rate  for  surfaee  elutter. 

Table  3-4  shows  the  performance  of  the  false-alarm  mitigation  teehniques  applied 
to  the  magnetometer  data  with  of  1  m.  The  table  gives  performance  at  each  stage  of 
the  algorithm:  magnetic  moment  direetion,  GoF,  and  depth  criterion.  The  confidence 
level  represents  the  overall  ranking,  0  being  the  highest,  6  being  the  lowest.  Only  the  first 
three  confidence  levels  are  shown  because  after  eonfidenee  level  2,  no  additional 
detections  are  recorded  until  the  magnetie  moment  orientation  filter  is  completely 
disengaged  (which  results  in  looking  at  the  full  data  set).  For  the  magnetometer  data,  all 
three  eonfidenee  levels  shown  are  substantially  better  than  with  no  false-alarm  mitigation. 
There  appears  to  be  no  real  improvement  across  these  three  confidence  levels  (see 
Figures  3-2  and  3-3).  There  is  performanee  improvement  when  the  first  stage  (moment 


^  There  was  a  small  set  of  surface  ordnance  emplaced  along  an  edge  of  the  MTR  for  this  test.  Since  the 
ordnance  was  placed  in  a  visible  pattern,  and  IDA  knew  before  evaluation  where  the  surface  ordnance 
was  located,  these  surface  items  were  excluded  from  the  baseline  and  from  the  sensor  data. 
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direction)  is  employed.  After  that  there  is  no  significant  improvement.  Given  that  the 
depth  restriction  used  in  this  assessment  (no  surface  ordnance)  is  site  specific,  the 
moment  direction  filter  is  the  most  robust  false-alarm  discriminator. 


Table  3-4.  Performance  versus  Stages  of  Algorithm  Implementation 
for  Magnetometer,  =  1  m 


Stage 

Confidence  Level  0 

Confidence  Level  1 

Confidence  Level  2 

Pc 

FAR  (m-^) 

Pc 

FAfl(m-2) 

Pc 

FAR  (m-2) 

Full  Sef 

0.726 

2.0  X  10-2 

0.726 

2.0  X  10-2 

0.726 

2.0  X  10-2 

Moment 

0.694 

9.2  X  10-2 

0.710 

9.9  X  10-2 

0.710 

1.1  X  10-2 

Fit 

0.581 

3.1  X  10-2 

0.677 

6.1  X  10-2 

0.710 

8.7x10-2 

Depth 

0.548 

2.6  X  10-2 

0.645 

5.2x10-2 

0.677 

7.4x10-2 

*  The  full  set  does  not  contain  any  mitigation.  Thus,  the  values  in  the  first  row  are  identical  for  each 
algorithm  confidence  ranking. 


Table  3-5  shows  the  performance  of  the  false-alarm  mitigation  techniques  applied 
to  the  magnetic  gradiometer  data.  The  approach  is  identical  to  that  used  for  the 
magnetometer.  For  the  magnetic  gradiometer  there  appears  to  be  no  measurable 
improvement  across  these  three  confidence  levels  (see  Figures  3-2  and  3-3).  Here,  there 
is  statistically  little  improvement  when  applying  the  moment  direction  filter.  The  same  is 
true  for  a  GoF  criterion. 

Table  3-5.  Performance  versus  Stages  of  Algorithm  Implementation  for 
Magnetic  Gradiometer,  Rc„t  =  1  m 


Stage  Confidence  Level  0  Confidence  Level  1  Confidence  Level  2 


Pc 

FAR  (m-2) 

Pc 

FAR  (m-2) 

Pc 

FAR  (m-2) 

Full  Sef 

0.710 

8.3  X  10-2 

0.710 

8.3x10-2 

0.710 

8.3x10-2 

Moment 

0.613 

3.9  X  10-2 

0.613 

4.2  X  10-2 

0.613 

4.5  X  10-2 

Fit 

0.403 

1.4  X  10-2 

0.468 

2.9  X  10-2 

0.532 

3.3x10-2 

Depth 

0.403 

1.2  X  10-2 

0.468 

2.2  X  10-2 

0.532 

2.6x10-2 

^  The  full  set  does  not  contain  any  mitigation.  Thus,  the  values  in  the  first  row  are  identical  for  each 
algorithm  confidence  ranking. 


The  strength  of  the  false-alarm  mitigation  technique  is  based  on  the  restriction  of 
the  direction  of  the  ordnance  magnetic  moment.  The  premise  of  this  technique  is  that  the 
net  magnetization  of  ordnance  is  dominated  by  that  magnetization  induced  by  the  geo¬ 
magnetic  field;  i.e.,  the  remanent  magnetization  is  small  (see  Section  2.2).  On  the  other 
hand,  the  source  of  a  false  alarm,  clutter,  is  predicted  to  exhibit  a  more  uniform  angular 
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distribution  of  magnetic  moments.  Figures  3-6(a)  and  3-7(a)  show  the  distribution  of 
magnetic  moment  direction  for  the  detected  ordnance  for  the  magnetometer  and  magnetic 
gradiometer,  respectively.  The  cutoff  for  the  algorithm  used  to  separate  moments  with  a 
confidence  level  of  2  or  less  is  approximately  30  deg.^  The  magnetometer  data  show  only 
1  out  of  45  with  a  relative  inclination  of  less  than  30  deg.  The  magnetic  gradiometer 
produces  a  larger  number  of  ordnance  having  a  relative  inclination  below  30  deg  (6  out  of 
44). 

Figures  3-6(b)  and  3-7(b)  show  the  distribution  of  the  relative  inclination  of  the 
false  alarms.  The  magnetometer  data  exhibit  a  significant  proportion  of  the  magnetic 
moment  angular  distribution  with  a  relative  inclination  below  30  deg  for  nonordnance. 
This  is  in  contrast  to  the  ordnance  population.  In  addition,  the  distribution  function  for 
false  alarms  from  the  magnetic  gradiometer  also  exhibits  a  significant  population  below  a 
relative  inclination  of  30  deg. 

3.1.4  Characterization  of  Missed  Ordnance 

Of  the  62  unique  targets  emplaced  at  the  Twentynine  Palms  MTR,  the  declaration 
set  produced  before  implementation  of  the  IDA  false-alarm  mitigation  algorithm  resulted 
in  the  detection  of  45  and  44  targets  using  the  magnetometer  and  magnetic  gradiometer, 
respectively,  at  a  critical  radius  of  1  m.  An  evaluation  of  the  undetected  targets  was 
performed  to  identify  any  trends  among  them.  Each  target  undetected  by  IDA  was 
reexamined  using  the  MTADS-DAS  at  the  target  location  listed  in  the  baseline.  For  each 
undetected  target,  a  fit  was  performed  around  the  baseline  location,  and  the  MTADS-DAS 
fitting  parameters  were  recorded.^  Based  on  this  analysis,  the  undetected  targets  can  be 
grouped  into  three  categories:  large  location  error,  weak  dipole  fits,  and  small  magnetic 
anomaly  signatures. 

1 .  Large  Location  Error.  The  magnetic  dipole  fits  provided  by  the  MTADS-DAS 
are  highly  dependent  on  the  size  of  the  declaration  box  (i.e.,  the  area  local  to 
the  anomaly  used  to  determine  the  fit  parameters).  For  the  magnetometer  and 
magnetic  gradiometer,  six  targets  and  four  targets,  respectively,  were  not 
classified  as  detections  because  MTADS-DAS  location  errors  were  between 


^  The  angle  used  is  measured  with  the  geo-coordinate  rotated  so  that  the  geomagnetic  field  is  along  the 
z-axis.  This  results  in  a  geomagnetic  field  inclination  of  90  deg.  Thus,  when  this  report  refers  to 
relative  inclination,  the  reference  is  the  rotated  geo-coordinate  system. 

^  This  could  not  be  done  for  two  ordnance  items  in  the  magnetic  gradiometer  set  because  neither  item 
was  surveyed  by  the  MTADS. 
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Figure  3-6.  Distribution  of  Magnetic  Moments  Relative  to  the  Geomagnetic  Field  as 
Measured  by  the  Magnetometer.  Relative  inclination  of  90  deg  is  aligned  with  the 
geomagnetic  field.  Relative  inclination  of  0  deg  is  orthogonal  to  geomagnetic  field. 

(a)  Ordnance,  (b)  False  alarms. 


Figure  3-7.  Distribution  of  Magnetic  Moments  Relative  to  the  Geomagnetic  Field  as 
Measured  by  the  Magnetic  Gradiometer.  Relative  inclination  of  90  deg  is  aligned 
with  the  geomagnetic  field.  Relative  inclination  of  0  deg  is  orthogonal  to 
geomagnetic  field,  (a)  Ordnance,  (b)  False  alarms. 
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1  m  and  2  m,  which  is  outside  the  l-m  critical  radius.  The  investigation  of  the 
largest  target,  a  2,000-lb  bomb,  provides  good  insight  into  the  idiosyncrasies 
of  the  dipole-fitting  algorithm.  Examination  of  this  target  revealed  that  the 
IDA  target  declaration  correctly  identified  the  ordnance’s  clearly  visible 
magnetic  anomaly.  But  the  location  from  the  fit  was  incorrect.  The  anomaly 
for  the  2,000-lb  bomb  was  fitted  using  four  different  declaration  boxes  with 
slightly  different  orientations,  centered  on  the  baseline  coordinate  of  the  ord¬ 
nance.  Only  one  of  these  orientations  provided  a  location  within  an  R^ri,  of 
1  m.  The  ordnance  items  discussed  here  were  not  classified  as  detections 
because  the  operator  was  unable  to  provide  an  optimal  fit  area  for  the  dipole 
fitting  algorithm. 

2.  Weak  Dipole  Fits.  The  small  (e.g.,  60-mm  mortar)  to  large  (e.g.,  250-lb 
bomb)  ordnance  items  were  often  obscured  by  the  clutter  background.  In  the 
presence  of  natural  or  anthropic  clutter,  the  MTADS-DAS  has  difficulty 
fitting  a  given  anomaly.  Two  targets  for  the  magnetometer  and  seven  for  the 
magnetic  gradiometer  were  not  detected  due  to  the  faint  dipole  signatures. 
These  correspond  to  small  targets  buried  at  depths  greater  than  1  m.  To 
identify  the  signatures  for  these  targets,  it  was  necessary  to  lower  the 
threshold  chosen  for  the  fit  to  ±20  y  for  this  investigation. 

3.  Small  Magnetic  Anomaly  Signatures.  Another  nine  (magnetometer)  and  seven 
(magnetic  gradiometer)  ordnance  items  were  missed  because  the  anomalies 
were  not  strong  enough  to  fit.  These  were  targets  obscured  by  the  cluttered 
background.  Perhaps  an  automated  procedure  set  at  an  appropriate 
threshold — for  instance,  using  multiple-box  orientations  to  optimize  GoF — 
would  fare  better.  Ultimately,  the  fitting  algorithm  is  limited  by  the  number 
of  data  points  taken  around  the  anomaly  and  the  signal-to-noise  ratio. 

Figures  3-8  through  3-10  show  the  detected  and  undetected  ordnance  as  a  function 
of  depth  for  the  three  sensors.  The  magnetometer  and  the  magnetic  gradiometer  both 
struggle  with  detection  of  the  deepest  ordnance.  This  is,  for  the  most  part,  a  result  of  the 
small  amplitude,  spatially  extended  signature  of  the  deep  ordnance.  Since  the  MTR  at 
Twentynine  Palms  has  a  relatively  high  level  of  geological  clutter,  these  broad  signatures 
can  be  lost  in  the  background.  On  the  other  hand,  for  smaller  ordnance  of  all  types,  the 
misses  tend  to  span  the  depths  of  emplacement. 

The  EMI  system  shows  a  more  compelling  trend.  Only  the  deepest  ordnance 
items  are  missed,  and  for  the  smaller,  near-surface  ordnance,  there  are  no  missed  items. 
This  suggests  that  the  EMI  system  is  very  robust  within  the  top  2  m,  where  no  ordnance 
was  missed. 
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Figure  3-8.  Type  of  Ordnance  versus  the  Depth  for  Detected  (Squares) 
and  Nondetected  (Stars)  for  the  Magnetometer 


Figure  3-9.  Type  of  Ordnance  versus  the  Depth  for  Detected  (Squares) 
and  Nondetected  (Stars)  for  the  Magnetic  Gradiometer 
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Figure  3-10.  Type  of  Ordnance  versus  the  Depth  for  Detected  (Squares) 
and  Nondetected  (Stars)  for  the  EMI  System 


3.1.5  False  Alarms 

There  is  a  measured  difference  between  the  depth  distribution  for  false  alarms  for 
the  active  EMI  system  and  the  passive  magnetostatic  detection  techniques.  Figure  3-11 
shows  the  number  of  false  alarms  as  a  function  of  depth  for  each  sensor.  Although  for  all 
sensors  there  are  a  substantial  number  of  false  alarms  in  the  first  0.1  m  of  the  ground,  the 
EMI  does  not  produce  many  additional  false  alarms  at  greater  depths  (only  4  out  of  197, 
or  approximately  2  percent  of  the  false  alarms  exist  below  1  m).  The  magnetometer  and 
magnetic  gradiometer  false-alarm  depth  distributions  have  approximately  4.3  and 
10  percent  of  their  respective  false-alarm  populations  at  depths  below  1  m.  Thus,  the 
depth  may  provide  some  method  for  discrimination,  but  a  number  of  issues  must  be 
addressed. 

1.  The  EMI  system  shows  poor  depth  accuracy  (larger  than  the  bins  used  to 
generate  Figure  3-11).  Thus,  the  clutter  responsible  for  the  EMI  anomaly  may 
have  a  different  depth  distribution  function  than  shown  in  Figure  3-11. 

2.  The  magnetometer  tends  to  perform  more  poorly  when  detecting  small  objects 
near  the  surface  (these  can  be  UXO)  than  the  EMI  system. 
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Figure  3-11.  The  Number  of  False  Alarms  as  a  Function  of  Fitted  Depth. 

(a)  Magnetometer,  (b)  Magnetic  Gradiometer.  (c)  EMI  System. 

3.2  MODELED  MAGNETIC  MOMENTS  AND  MOMENT  DIRECTION 

The  magnetic  moment'^  for  each  ordnance  item  emplaced  at  the  Twentynine 
Palms  MTR  has  been  modeled.  Appendix  A  gives  a  table  of  the  model  moments  for  each 
ordnance  item.  The  model  assumes  that  the  ordnance  item  can  be  approximated  by  a 
prolate  spheroid  of  the  same  dimensions  as  the  ordnance.  For  this  single  case,  only 
induced  magnetization  is  considered.  For  this  data  set,  the  calculation  assumes  that  the 
geomagnetic  field  has  an  inclination  of  approximately  60  deg  and  a  declination  of 
approximately  14  deg,  and  that  the  magnitude  of  the  geomagnetic  field  is  0.497  Oe.  The 
relative  magnetic  permeability  is  assumed  to  be  100. 

The  modeled  magnetic  moment  (also  called  the  volume  magnetization)  for  all 
ordnance  items  is  then  rotated  into  the  geomagnetic  coordinate  system  (see  Figure  2-1)  to 
permit  comparison  with  the  magnetic  moment  of  all  detected  targets  determined  using  the 


Here,  the  magnetic  moment  is  in  units  of  A-m^.  The  difference  between  this  and  the  magnetic  moment 
often  used  is  a  factor  of  go  (4k  x  10'^  H/m). 
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MTADS-D AS.  For  this  model  the  ordnance  item  is  assumed  to  be  a  solid  ferrous  object. 
Real  ordnance  is  hollow  (with  respect  to  the  ferrous  material)  and  contains  breaks  in  the 
ferrous  shell  for  the  fuze,  etc.  In  addition,  although  the  false-alarm  mitigation  technique 
assumes  that  the  remanent  magnetization  is  small,  it  is  not  assumed  to  be  zero.  A  small 
remanent  magnetization  (even  only  20  percent  of  the  maximum  geomagnetically  induced, 
which  is  the  assumption  for  confidence  level  2)  can,  in  specific  instances,  measurably 
alter  the  magnitude  and  direction  of  the  net  magnetization.  Therefore,  the  magnetic 
moment  given  in  Appendix  A  provides  only  a  starting  point  for  estimating  signatures. 

3.2.1  Comparison  with  Fit  Magnetic  Moments  from  the  Magnetometer  Data 

Figure  3-12(a)  shows  a  comparison  of  the  modeled  magnetic  moment  and  the  fit 
magnetic  moment  for  all  individual  ordnance  items  detected  by  the  magnetometer. 
Although  there  appears  to  be  correlation  between  the  modeled  and  fit  moments,  the  linear 
correlation  coefficient  is  only  0.72.  To  understand  the  relative  lack  of  strong  correlation 
between  the  modeled  magnetic  moment  and  the  fit  magnetic  moment,  it  is  valuable  to 
investigate  how  other  parameters,  such  as  GoF  and  ordnance  type,  influence  the 
correlation. 

Figure  3- 12(a)  includes  detected  ordnance  with  a  poor  GoF.  For  a  dipole  fit  an 
acceptable  GoF  is  a  fit  greater  than  90  percent.''  Figure  3- 12(b)  is  the  GoF  as  a  function 
of  the  ratio  of  the  fit  magnetic  moment  (determined  by  the  MTADS-D  AS)  to  the  modeled 
magnetic  moment,  referred  to  as  the  magnetic  moment  ratio.  It  shows  that  there  are  five 
ordnance  items  with  magnetic  moment  ratios  greater  than  3.5;  four  of  these  are  60-mm 
mortars.  Although  these  appear  to  be  statistical  outliers,  they  are  more  likely  caused  by 
the  specific  magnetic  history,  emplacement  history,  or  size  and  orientation  of  these 
ordnance  items.  All  these  items  have  good  dipole  fits.  Figure  3- 12(c)  shows  the  ratio  of 
magnetic  moments  as  a  function  of  modeled  magnetic  moment.  This  figure  shows  that 
four  (two  points  overlap)  items  with  the  smallest  modeled  magnetic  moments  (these  are 
60-mm  mortars  in  the  same  physical  orientation)  also  have  largest  magnetic  moment 
ratio. 

Figure  3- 13(a)  is  a  histogram  of  the  distribution  of  the  magnetic  moment  ratio. 
The  mean  magnetic  moment  ratio  is  1.40  with  a  standard  deviation  of  1.17.  This  mean 


"  This  level  of  fit  was  chosen  on  advice  of  one  of  the  developers  of  the  MTADS-DAS,  Dr.  Bruce  Barrow, 
AETC,  Inc.  The  GoF  uses  the  Levenberg-Marquardt  method.  The  scale  for  GoF  runs  from  0.0  to  1.0. 
The  figures  present  the  GoF  using  a  scale  from  0  to  100. 


3-17 


(a)  (b) 


10'^  10°  10^ 

Modeled  Magnetic  Moment  (A-m^) 

Figure  3-12.  Scatter  Plots  of  the  Modeled  Magnetic  Moment  and  the  Fitted  Magnetic 
Moment  (from  the  MTADS-DfiiS)  for  Magnetometer  Data,  (a)  Modeled  Magnetic  Moment 
versus  the  Fitted  Magnetic  Moment,  (b)  Ratio  of  Magnetic  Moment  versus  the  GoF. 

(c)  Ratio  of  Magnetic  Moment  versus  the  Modeled  Magnetic  Moment.  Note  that 
the  largest  magnetic  moment  ratio  corresponds  to  the  smallest  moments  (all  60-mm 
mortars).  At  the  other  extreme,  the  small  magnetic  moment  ratio  corresponds 
to  the  largest  moment,  250-lb  bombs. 

magnetic  ratio  cannot  be  accounted  for  by  assuming  a  hollow  ordnance  item  because  the 
effect  of  a  hollow  object  would  reduce  only  slightly  the  magnitude  of  the  real  moment 
relative  to  the  modeled  moment  for  a  solid  prolate  spheroid  (resulting  in  a  mean  magnetic 
moment  ratio  less  than  1)  (Altshuler,  1996;  Nelson  et  al.,  1998).  On  the  other  hand,  a 
small  remanent  contribution  to  the  net  magnetization  can  cause  this  increase  with  the 
observed  moment  orientation  distribution  function. 

There  are  also  three  ordnance  items  with  a  magnetic  moment  ratio  of  less  than 
1/3.  Two  of  these  are  in  the  0-0.25  bin;  the  third  is  in  the  0.25-0.5  bin  (see  Figure  3-13). 
These  are  far  from  the  modeled  moments.  These  ordnance  items  are  all  500-lb  bombs. 
This  suggests  that  the  size  or  another  parameter  used  to  model  the  magnetic  moment  of 
the  500-lb  bomb  is  not  accurate.  Figure  3-13(b)  shows  the  mean  magnetic  moment  ratio 


In  fact,  seven  of  nine  of  the  smallest  magnetic  moment  ratios  arc  500-lb  bombs. 
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Figure  3-13.  (a)  Histogram  of  the  Number  of  Ordnance  Items  as  a  Function  of  Ratio  of 
Magnetic  Moment.  The  bin  size  is  0.25.  (b)  Mean  Magnetic  Moment  Ratio  for  the 
Seven  Different  Types  of  Ordnance  Detected  on  the  MTR.  Both  the 
500-lb  bombs  and  the  60-mm  mortars  are  substantially 
different  from  the  other  ordnance  items. 

as  a  function  of  the  type  of  ordnance.  Having  a  magnetic  moment  ratio  very  different 
from  1,  the  60-mm  mortar  and  the  500-lb  bomb  show  substantial  differences  from  the 
model.  Table  3-6  lists  characteristics  of  the  ordnance  items  that  exhibit  the  largest 
discrepancy  from  a  magnetic  moment  ratio  of  1 .  Although  one  of  the  mortars  has  a  large 
relative  depth  error,  the  emplacement  depth  is  only  30  cm.  Thus,  the  absolute  error  is  less 
than  16  cm.  On  the  other  hand,  there  are  large  location  and  depths  errors  for  two  of  the 
three  500-lb  bombs.  The  two  largest  relative  depth  errors  for  these  are  a  result  of  greater 
than  1 .25-m  depth  errors.  The  remaining  500-lb  bombs  detected  have  a  mean  depth  error 
of  less  than  25  cm.  The  last  500-lb  bomb  has  a  relatively  poor  GoF,  83  percent,  the 
poorest  for  all  500-lb  bombs.  For  the  500-lb  bombs,  it  appears  that  these  three  poor 
detections  are  anomalies. 
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Table  3-6.  Fit  Properties  of  Ordnance  Not  Consistent  with 
Magnetic  Moment  Model  for  Magnetometer 


Ordnance 

Type* 

Modeled 

Moment 

(A-m^) 

Fit 

Moment 

(A-m") 

Moment 

Ratio 

Relative 

Depth 

Error** 

Radial 

Error 

(m) 

GoF 

(percent) 

60-mm  mortar 

0.03 

0.14 

4.67 

0.53 

0.02 

98 

60-mm  mortar 

0.03 

0.14 

4.67 

0.17 

0.09 

96 

60-mm  mortar 

0.03 

0.13 

4.33 

0.03 

0.02 

92 

60-mm  mortar 

0.03 

0.12 

4.00 

0.10 

0.08 

97 

500-lb  bomb 

65.7 

6.0 

0.09 

0.42 

0.90 

96 

500-lb  bomb 

35.2 

3.8 

0.11 

0.35 

0.72 

91 

500-lb  bomb 

68.6 

18.53 

0.27 

0.06 

0.70 

83 

*  Ordnance  listed  in  italics  are  also  detected  by  the  magnetic  gradiometer  with  poor  correspondence  with 
the  modeled  signature. 

The  relative  depth  error  is  the  ratio  of  the  magnitude  of  the  difference  between  the  emplaced  depth  and 
the  fit  depth  to  the  emplaced  depth. 

3.2.2  Comparison  with  Fit  Magnetic  Moment  Direction  from 
Magnetometer  Data 

We  compare  the  fit  magnetic  moment  direction  (inclination  and  declination)  to 
the  modeled  magnetic  moment  direction.  To  simplify  the  comparison,  inclination  and 
declination  are  converted  to  the  relative  inclination  discussed  in  Sections  2.2  and  3.1.3. 
Figure  3-14  shows  the  comparison  of  the  relative  inclinations.  The  relative  inclination  for 
the  fitted  and  modeled  moments  shows  virtually  no  correlation,  which  suggests  that  a 
small  remanent  magnetization  should  be  added  to  the  net  magnetization  of  the  ordnance. 
Doing  so  will  alter  the  direction  of  the  magnetic  moment,  at  times  significantly.  In 
addition,  there  is  potential  that  the  MTADS-DAS  does  not  robustly  fit  the  direction  of  the 
magnetic  moment  in  a  noisy  environment.  Still,  the  general  conclusion  from  the  model — 
that  the  majority  of  ordnance  should  have  magnetic  moment  orientations  that  are  prefer¬ 
entially  directed  along  the  geomagnetic  field — appears  to  be  supported  by  the  distribution 
of  ordnance  as  a  function  of  relative  inclination  (see  Figure  3-6). 

3.2.3  Comparison  with  Fit  Magnetic  Moments  from  Magnetic 
Gradiometer  Data 

Figure  3- 15(a)  shows  a  comparison  of  the  modeled  magnetic  moment  and  the 
fit  magnetic  moment  for  all  individual  ordnance  items  detected  by  the  magnetic 
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Figure  3-14.  Scatter  Plot  of  the  Relative  inclination  for  the  Modeled  Magnetic  Moment 
versus  that  of  the  Fitted  Magnetic  Moment.  There  is  minimal  correlation 
between  the  model  and  the  fitted  data. 

gradiometer.  There  appears  to  be  less  correlation  between  the  modeled  and  fit  moments 
than  in  the  case  of  the  magnetometer.  For  the  magnetic  gradiometer  data  the  linear 
correlation  coefficient  is  0.54.  One  reason  for  this  might  be  the  decrease  in  spatial 
sampling  for  the  magnetic  gradiometer  system,  which  is  approximately  6  cm  along  track 
but  only  0.5  m  cross  track,  compared  to  0.25  m  cross  track  for  the  magnetometer 
configuration.  Figure  3- 15(b)  shows  the  GoF  as  a  function  of  the  ratio  of  the  fit  magnetic 
moment  to  the  modeled  magnetic  moment.  Because  the  GoF  for  the  magnetic  gradiom¬ 
eter  data  is  poorer  than  that  of  the  magnetometer  data,  it  is  difficult  to  draw  strong 
conclusions.  But  for  this  data,  some  of  the  poorest  fits  appear  to  have  either  high  ratios 
(greater  than  3)  or  low  ratios  (less  than  1/3).  Figure  3- 15(c)  shows  the  ratio  of  magnetic 
moments  as  a  function  of  modeled  magnetic  moment.  As  with  the  magnetometers,  the 
60-mm  mortars  and  500-lb  bombs  have  the  largest  and  smallest  magnetic  moment  ratios, 
respectively.  Still,  there  are  a  number  of  poorly  modeled  moments.  For  example,  there  is 
an  8-in.  rocket'^  with  a  magnetic  moment  ratio  of  6.64.  But  this  rocket  has  a  GoF  of  only 
66  percent  and  has  substantial  location  errors  (a  2.23-m  depth  error  and  a  0.88-m  location 
error)  and  thus  appears  to  be  an  outlier  relative  to  the  other  8-in.  rockets. 

This  rocket  was  not  detected  in  the  magnetometer  data  analysis. 
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Figure  3-15.  Scatter  Plots  of  the  Modeled  Magnetic  Moment  and  the  Fitted  Magnetic 
Moment  (from  the  MTADS-DAS)  for  Magnetometer  Data,  (a)  Modeled  Magnetic  Moment 
versus  the  Fitted  Magnetic  Moment,  (b)  Ratio  of  Magnetic  Moment  versus  the  GoF. 

(c)  Ratio  of  Magnetic  Moment  versus  the  Modeled  Magnetic  Moment.  Note  that 
in  general  the  largest  magnetic  moment  ratios  correspond  to  the  smallest  moments 
(all  60-mm  mortars).  At  the  other  extreme,  the  small  magnetic  moment 
ratios  correspond  to  the  largest  moment,  250-lb  bombs. 

Figure  3- 16(a)  is  a  histogram  of  the  distribution  of  the  magnetic  moment 
ratio,  The  mean  magnetic  moment  ratio  is  1.37,  with  a  standard  deviation  of  1.39. 
Figure  3- 16(b)  shows  the  mean  magnetic  moment  ratio  for  each  type  of  ordnance.  It  is 
apparent  from  this  figure  that,  as  with  the  magnetometer  data,  the  magnetic  gradiometer 
fit  moment  deviates  from  the  model  most  extensively  for  the  60-mm  mortars  and  500-lb 
bombs.  In  addition,  the  8-in.  rocket  also  deviates  from  a  mean  ratio  of  1.  But  if  the  outlier 
discussed  above  is  not  counted,  the  mean  magnetic  moment  ratio  for  8 -in.  rockets  drops 
to  1.41. 

Table  3-7  lists  ordnance  with  magnetic  moment  ratios  that  are  either  greater  than 
three  or  less  than  1/3.  The  entries  in  italics  are  also  included  in  Table  3-6  (for  the 
magnetometer).  Of  the  three  60-mm  mortars  in  common  with  the  magnetometer  set. 
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Figure  3-16.  (a)  Histogram  of  the  Number  of  Ordnance  as  a  Function  of  Ratio  of  Magnetic 
Moment.  The  bin  size  is  0.25.  (b)  Mean  Magnetic  Moment  Ratio  for  the  Seven  Different 
Types  of  Ordnance  Detected  on  the  MTR.  The  500-lb  bombs,  8-in.  rockets,  and 
60-mm  mortars  are  substantially  different  from  the  other  ordnance  items. 


Table  3-7.  Fit  Properties  of  Ordnance  Not  Consistent  with 
Magnetic  Moment  Model  for  Magnetic  Gradiometer 


Ordnance 

Type* 

Modeled 

Moment 

(A-m^) 

Fit 

Moment 

(A-m^) 

Moment 

Ratio 

Relative 

Depth 

Error** 

Radial 

Error 

(m) 

GoF 

(percent) 

60-mm  mortar 

0.03 

0.15 

4.71 

0.13 

0.11 

87 

60-mm  mortar 

0.03 

0.13 

4.08 

0.07 

0.11 

45 

60-mm  mortar 

0.03 

0.13 

3.14 

0.03 

0.13 

89 

155-mm  proj. 

2.6 

0.8 

0.31 

0.02 

0.08 

97 

8-in.  rocket 

9.1 

60.2 

6.64 

0.96 

0.88 

66 

8-in.  rocket 

7.98 

26.7 

3.35 

0.23 

0.16 

89 

500-lb  bomb 

43.8 

8.2 

0.19 

0.37 

0.93 

40 

500-lb  bomb 

65.7 

18.15 

0.28 

0.19 

0.48 

85 

*  Ordnance  listed  in  italic  are  also  detected  by  the  magnetometer  with  poor  correspondence  with  the 
modeled  signature. 


The  relative  depth  error  is  the  ratio  of  the  magnitude  of  the  difference  between  the  emplaced  depth  and 
the  fit  depth  to  the  emplaced  depth. 
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two  have  relatively  good  fits,  while  one  has  a  very  poor  fit.  The  trend  for  the  500-lb 
bombs  is  slightly  different.  The  nonitalicized  entry  was  not  detected  from  the  magne¬ 
tometer  data  and  has  a  very  poor  GoF.  On  the  other  hand,  the  target  that  is  common  to 
both  the  magnetometer  and  magnetic  gradiometer  set  has  very  different  characteristics. 
The  fitted  magnetic  moment  for  this  bomb  using  the  magnetic  gradiometer  data  is  three 
times  that  fitted  from  the  magnetometer  data. 

3.2.4  Comparison  with  Fit  Magnetic  Moment  Direction  from 

Magnetic  Gradiometer  Data 

A  comparison  of  the  relative  inclinations  for  the  modeled  magnetic  moment  and 
the  fit  magnetic  moment  for  the  magnetic  gradiometer  shows  a  trend  similar  to  the 
magnetometer.  Again,  the  relative  inclination  for  the  fitted  and  modeled  moments  shows 
virtually  no  correlation.  The  general  conclusion  from  the  model — that  the  majority  of 
ordnance  should  have  magnetic  moment  orientations  that  are  preferentially  directed  along 
the  geomagnetic  field — appears  to  be  supported  by  the  distribution  of  ordnance  as  a 
function  of  relative  inclination  [see  Figure  3-7(a)]  but  not  as  strongly.  This  is  most  likely 
caused  by  the  poorer  fit  quality  of  the  magnetic  gradiometer  data. 

3.2.5  Comparison  of  Fit  Magnetic  Moments  Magnitude  and  Direction 

Between  the  Magnetometer  and  Magnetic  Gradiometer 

To  understand  the  robustness  of  the  algorithm  used  to  do  false-alarm  mitigation,  it 
is  important  to  know  the  accuracy  of  the  fit  algorithms  used  by  the  MTADS-DAS.  One 
way  to  test  the  accuracy  is  to  compare  the  fit  algorithm  for  the  magnetometer  data  with 
that  for  the  magnetic  gradiometer  data.  Figure  3- 17(a)  shows  the  magnetic  moment  ratios 
for  ordnance  common  to  both  the  magnetometer  and  magnetic  gradiometer.  The  data 
exhibit  a  trend,  but  with  some  scatter,  which  results  in  a  linear  correlation  coefficient  of 
approximately  0.58.  There  are  a  couple  of  common  ordnance  items  which  show  large 
magnetic  moment  ratios  (and  a  couple  that  show  small  ratios)  for  both  sensors.  Table  3-8 
lists  characteristics  of  the  ordnance.  The  data  show  that  moments  anomalously  different 
from  the  modeled  moment  (i.e.,  greater  or  less  than  the  modeled  moment  by  a  factor  of  2 
or  1/2,  respectively)  are  fairly  similar  for  both  sensors. 

On  the  other  hand,  the  primary  parameter  used  for  discrimination  is  the  relative 
inclination.  Figure  3-17(b)  shows  the  relative  inclination  for  ordnance  common  to  both 
the  magnetometer  and  magnetic  gradiometer.  With  a  linear  correlation  coefficient  of 
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Figure  3-17.  (a)  Scatter  Plot  of  the  Magnetic  Moment  Ratio  for  Common  Targets 
Between  the  Magnetometer  and  Magnetic  Gradiometer  Data  Sets. 

(b)  Scatter  Plot  of  the  Relative  Inclination  for  Common  Targets 
Between  the  Magnetometer  and  Magnetic  Gradiometer  Data  Sets. 


Table  3-8.  Fit  Properties  Not  Consistent  with  Magnetic  Moment  Model  for  Either 
Magnetometer  or  Magnetic  Gradiometer  Data  Common  to  Both  Data  Sets 


Ordnance 

Type 

Moment 

Ratio 

Mag. 

Moment 

Ratio 

Grad. 

Depth 
Error  (m) 
Mag. 

Depth 
Error  (m) 
Grad. 

Radial 
Error  (m) 
Mag. 

Radial 
Error  (m) 
Grad. 

60-mm  mortar 

4.0 

4.71 

0.03 

0.04 

0.09 

0.11 

60-mm  mortar 

4.67 

4.08 

0.16 

0.09 

0.02 

0.11 

60-mm  mortar 

4.67 

3.14 

0.03 

0.13 

0.08 

0.0 

60-mm  mortar 

4.33 

2.51 

0.01 

0.12 

0.02 

0.13 

155-mm  proj. 

0.56 

0.31 

0.0 

0.01 

0.16 

0.4 

8-in.  rocket 

2.32 

3.35 

0.4 

0.5 

0.36 

0.16 

500-lb  bomb 

0.09 

0.28 

1.64 

0.74 

0.90 

0.48 

0.51,  these  data  do  not  show  a  strong  trend.  Table  3-9  lists  the  ordnance  items  that  have 
relative  inclination  less  than  the  approximate  30-deg  cutoff  used  to  sort  the  data.  Five  of 
the  seven  ordnance  items  are  detected  by  both  sensors.  Although  none  of  these  items 
exhibit  anomalously  variant  magnetic  moment  ratios,  the  differences  in  relative  inclina¬ 
tion  can  be  substantial.  Only  target  ID  number  1025  (an  8-in.  rocket)  is  labeled  with  a 
low  confidence  level  in  both  data  sets  because  of  the  relative  inclination. 
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Table  3-9.  Ordnance  with  Small  Relative  Inclination  for  Either  the 
Magnetometer  or  Magnetic  Gradiometer 


Magnetometer 

Magnetic  Gradiometer 

Modeled  Moment 

Target 

ID 

Magnetic 

Moment 

(A-m") 

Relative 

Inclination 

(deg) 

Magnetic 

Moment 

(A-m^) 

Relative 

Inclination 

(deg) 

Magnetic 

Moment 

(A-m^) 

Relative 

Inclination 

(deg) 

1025 

4.7 

21.4 

4.8 

-16.8 

3.5 

57.1 

1060 

0.12 

67.22 

0.19 

0.34 

0.11 

84.43 

1044 

0.95 

36.2 

1.43 

21.3 

0.95 

48.78 

1015 

— 

— 

3.33 

25.6 

7.74 

47.86 

1032 

18.52 

64.7 

26.71 

26.2 

7.98 

65.5 

1031 

6.00 

44.4 

6.67 

27.61 

7.98 

65.5 

1011 

25.85 

36.44 

— 

— 

50.24 

75.89 

3.3  SENSOR  PERFORMANCE 

This  section  characterizes  the  system  location  performance  for  detected  ordnance 
by  evaluating  the  MTADS-DAS  parameters  and  determining  trends.  For  each  sensor  type, 
the  entire  declaration  set  (the  full  set)  is  matched  to  emplaced  ordnance.  A  critical  radius 
of  1  m  is  used  to  determine  the  set  of  declarations  that  match  emplaced  ordnance  in  the 
target  baseline.  The  magnetometer  declarations  match  to  45  emplaced  ordnance  items, 
the  magnetic  gradiometer  declarations  match  to  44  emplaced  ordnance  items,  and  the 
EMI  sensor  declarations  match  to  51  emplaced  ordnance  items  within  the  1-m  critical 
radius. 

3.3.1  Sensor  Performance:  Radial  Location  Accuracy 

For  each  sensor  declaration  which  matches  an  emplaced  ordnance  item  within  the 
1-m  critical  radius,  the  coordinates  of  the  fitted  anomaly  and  radial  location  accuracy  or 
radial  location  error  are  recorded  (radial  location  accuracy  =  fitted  anomaly  location  - 
actual  location  of  detected  ordnance  item).  The  results  of  the  analysis  follow  below. 

3.3.1.1  Magnetometer 

The  magnetometer  data  set  results  in  the  detection  of  35  ordnance  items  within  a 
0.5-meter  and  45  targets  within  a  1-m  The  mean  radial  location  accuracy  is 
0.30  m  with  a  standard  deviation  of  0.27  m  (see  Table  3-3).  Figures  3-18(a)-(d)  show  a 
series  of  scatter  plots  for  different  fit  and  modeled  parameters  for  the  magnetometer  data. 


3-26 


(a)  (b) 


0  0.5  1  0  0.5  1 


Radial  Location  Error  (m) 


Figure  3-18.  Radial  Location  Error  for  the  Magnetometer,  (a)  Versus  Modeled  Magnetic 
Moment,  (b)  Versus  GoF.  (c)  Versus  Depth,  (d)  Versus  Depth  Error. 

There  is  minimal  correlation  between  magnetic  moment  and  radial  location  error:  the 
position  accuracy  for  the  various  size  objects  is  roughly  equivalent.  There  is  some  corre¬ 
lation  between  radial  location  accuracy  and  GoF;  of  the  10  targets  with  radial  location 
error  greater  than  0.5  m,  4  anomalies  have  fits  less  than  85  percent,  but  the  6  additional 
anomalies  with  large  location  errors  have  fits  greater  than  90  percent.  Therefore,  we 
cannot  say  definitively  that  those  targets  with  poor  fits  are  responsible  for  decreasing  the 
overall  location  accuracy. 

A  weak  trend  exists  for  increased  radial  error  to  be  associated  with  increased 
depth.  Finally,  larger  radial  errors  for  targets  appear  to  correlate  with  large  depth  errors, 
but  the  correlation  is  not  strong.  Note  that  there  is  no  correlation  between  the  sign  of  the 
depth  error  and  the  radial  location  accuracy. 

3.3.1. 2  Magnetic  Gradiometer 

The  magnetic  gradiometer  detects  36  teirgets  within  a  0.5-m  and  44  targets 
within  a  1-m  The  mean  radial  location  accuracy  is  0.32  m  with  a  standard  deviation 
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of  0.24  m  (see  Table  3-3).  Figures  3-19(a)-(d)  show  a  series  of  scatter  plots  for  different 
fit  and  modeled  parameters  for  the  magnetic  gradiometer  data.  As  with  the  magne¬ 
tometer,  the  magnetic  gradiometer  location  accuracy  is  weakly  correlated  at  best  to  the  fit 
magnetic  moment.  On  the  other  hand,  the  GoF  is  slightly  more  correlated  to  the  location 
accuracy:  the  poorer  location  accuracies  correlate  with  the  poorer  fits.  For  GoF  greater 
than  90  percent,  radial  errors  are  generally  less  than  0.4  m.  There  is  a  trend  for  higher 
radial  error  to  be  associated  with  anomalies  that  have  a  GoF  less  than  90  percent. 
The  sensor’s  location  accuracy  shows  weak  correlation  with  object  depth.  There  is  also 
weak  correlation  between  radial  location  accuracy  and  depth  error.  Again,  there  appears 
to  be  no  correlation  with  the  sign  of  the  depth  error.  For  radial  error  less  than  0.5  m,  the 
normalized  depth  error  is  generally  less  than  50  percent. 


Radial  Location  Error  (m) 

Figure  3-19.  Radial  Location  Error  for  the  Magnetic  Gradiometer.  (a)  Versus  Modeled 
Magnetic  Moment,  (b)  Versus  GoF.  (c)  Versus  Depth,  (d)  Versus  Depth  Error. 

3.3.1.3  EMI  Sensor 

The  EMI  sensor  exhibits  very  good  success  detecting  the  emplaced  ordnance. 
Forty-seven  targets  are  detected  within  the  0.5-m  The  mean  radial  location  accuracy 
is  0.22  m  with  a  standard  deviation  of  0.16  m  (see  Table  3-3).  Figure  3-20  shows  a  series 
of  scatter  plots  for  different  fit  parameters  for  the  EMI  data.  The  EMI  system  has  better 


3-28 


location  accuracy  than  either  the  magnetometer  or  the  magnetic  gradiometer.  There  is  no 
noticeable  trend  between  location  accuracy  and  size  (either  ferrous  or  nonferrous).  Most 
targets  which  have  a  GoF  greater  than  80  percent  have  a  location  error  less  than  0.2  m. 
Again,  as  with  both  the  magnetometer  and  the  magnetic  gradiometer,  only  the  depth  error 
(magnitude)  appears  to  be  correlated  with  the  radial  accuracy,  and  this  weakly. 


Radial  Location  Error  (m) 


Figure  3-20.  Radial  Location  Error  for  the  EMI  System,  (a)  Versus  Modeled  Magnetic 
Moment,  (b)  Versus  GoF.  (c)  Versus  Depth,  (d)  Versus  Depth  Error. 

The  overall  location  accuracy  for  all  sensors  is  quite  good,  and  the  EMI  sensor 
data  show  especially  good  location  accuracy.  Ultimately,  though,  there  is  no  compelling 
trending  of  location  data  with  the  other  MTADS-DAS  fit  parameters. 


3.3.2  Sensor  Performance:  Depth,  Depth  Accuracy 

For  each  sensor  declaration  which  matched  an  emplaced  target  within  the  1-m 
critical  radius,  the  fitted  depth  and  depth  error  (depth  error  =  fitted  depth  -  actual  depth  of 
the  emplaced  target)  is  recorded.  The  results  of  our  analysis  are  discussed  below. 

Since  the  baseline  items  are  emplaced  at  realistic  depths — large  objects  are 
typically  emplaced  at  depths  deeper  than  the  smaller  ordnance  items — an  analysis  of  the 
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trends  for  each  sensor  provides  physical  insight  into  the  sensor’s  ability  to  detect 
ordnance  of  various  sizes  and  at  various  depths.  For  all  three  sensor  types,  the  fitted  depth 
of  the  ordnance  is  fairly  well  correlated  to  the  fitted  moment;  for  the  EMI  system,  the 
fitted  depth  is  also  correlated  to  nonferrous  size.  This  is  consistent  with  the  emplacement 
protocol. 

3.3.2.1  Magnetometer 

The  average  magnetometer  depth  accuracy*'*  for  the  entire  set  of  targets  is  approx¬ 
imately  0.23  m,  with  a  standard  deviation  of  0.32  m.  Of  the  45  declarations  that  matched 
baseline  ordnance,  the  magnetometer  detected  33  targets  with  a  depth  accuracy  of  less 
than  0.25  m,  40  targets  with  a  depth  accuracy  of  less  than  0.5  m,  and  all  but  two  targets 
within  1.0  m.  These  final  two  are  more  than  three  standard  deviations  away  from  the 
mean.  Removing  the  outliers  results  in  a  mean  depth  error  of  0.17  m,  with  a  standard 
deviation  of  0.18  m.  As  shown  in  Figure  3-21  (a),  the  depth  error  is  not  correlated  with  the 
fit  magnetic  moment  of  the  object.  In  fact.  Figures  3-21(b)-(d)  show  that  the  depth 
accuracy  is  independent  of  depth  and  is  only  weakly  correlated  to  the  GoF  parameters.  Of 
the  45  targets,  9  targets  have  a  GoF  less  than  90  percent;  it  is  important  to  note,  never¬ 
theless,  that  poor  fits  do  not  correlate  with  poor  depth  accuracy  for  this  sensor. 

3.3.2.2  Magnetic  Gradiometer 

Given  the  poor  fit  of  the  model  used  in  the  MTADS-DAS,  the  magnetie  gradiom¬ 
eter  has  surprisingly  similar  depth  accuracy  to  that  of  the  magnetometer  [see  Figures 
3-22(a)-(d)].  The  average  depth  aecuracy  for  the  entire  set  of  targets  is  approximately 
0.26  m,  with  a  standard  deviation  of  0.37  m.  There  are  29  targets  located  within  0.25  m, 
37  targets  within  0.5  m,  43  targets  within  1  m,  and  1  outlier  at  2  m.  This  last  ordnance 
item,  a  250-lb  bomb  with  a  depth  error  of  2.23  m,  is  actually  close  to  another  bomb, 
which  causes  the  error  in  estimation  of  the  depth.  By  removing  this  ordnance  item  from 
the  baseline,  the  mean  depth  accuraey  improves  to  0.22  m,  with  a  standard  deviation  of 
0.21  m.  In  general,  depth  aceuracy  for  the  magnetic  gradiometer  is  poorly  correlated  to 
the  magnitude  of  the  magnetic  moment  [see  Figure  3-22(a)].  There  is  also  poor  correla¬ 
tion  between  the  fitted  depth  and  the  depth  accuracy. 


*'*  We  refer  to  the  depth  accuracy  as  the  magnitude  of  difference  between  the  emplaced  depth  and  fitted 
depth.  The  depth  error  is  the  true  difference. 
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Figure  3-21.  Depth  and  Depth  Error  for  the  Magnetometer,  (a)  Depth  Error 
versus  Modeled  Magnetic  Moment,  (b)  Depth  Error  versus  GoF. 

(c)  Depth  Error  versus  Depth,  (d)  Depth  versus  GoF. 
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Figure  3-22.  Depth  and  Depth  Error  for  the  Magnetic  Gradiometer.  (a)  Depth  Error 
versus  Modeled  Magnetic  Moment,  (b)  Depth  Error  versus  GoF. 

(c)  Depth  Error  versus  Depth,  (d)  Depth  versus  GoF. 
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3.3.2.3  EMI  Sensor 


The  EMI  system  exhibits  trends  similar  to  the  other  two  sensors.  In  general, 
the  depth  and  depth  accuracy  are  poorly  correlated  to  the  other  fit  parameters  [see 
Figures  3-23(a)-(d)].  The  GoF  for  most  ordnance  is  the  lowest  of  all  the  sensors.  The 
EMI  system  shows  a  systematic  misestimation  of  depth  (a  mean  undererestimation  of 
0.18  m).  The  average  depth  error  is  the  largest  for  all  three  sensors:  approximately  0.33  m 
with  a  standard  deviation  of  0.36  m.  The  EMI  located  44  targets  within  a  depth  of  0.5  m, 
with  7  additional  ordnance  items  at  depths  greater  than  0.5  m.  As  with  the  two  other 
sensors,  there  are  a  number  of  detections  that  are  more  than  three  standard  deviations 
from  the  mean  magnitude  of  the  depth  error.  Removing  these  and  recalculating  the  mean 
magnitude  of  the  depth  error  yields  0.27  m,  with  a  standard  deviation  of  0. 19  m. 
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Figure  3-23.  Depth  and  Depth  Error  for  the  EMI.  (a)  Depth  Error  versus  Nonferrous  Size. 

(b)  Depth  Error  versus  GoF.  (c)  Depth  Error  versus  Depth,  (d)  Depth  versus  GoF. 

3.4  DETECTIONS  AND  FALSE  ALARMS  COMMON  TO 
MAGNETOMETER  AND  EMI 

An  analysis  of  the  declarations  common  to  the  magnetometer  and  the  EMI 
systems  was  performed  to  understand  the  general  trends  associated  with  detection  by 
multiple  systems.  This  section  contains  a  comparison  of  the  MTADS-DAS  fit  parameters 
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for  this  sensor  pair.  A  common  detection  is  declared  for  all  ordnance  matched  to  a 
declaration  of  both  sensors  in  the  pair.  A  common  false  alarm  is  declared  for  any  false 
alarms  within  1 .0  m  of  each  other.  The  magnetometer  and  EMI  pair  of  sensors  is  referred 
to  as  ME. 

3.4.1  Common  Detections 

All  declarations  within  a  1.0-m  /?„„  of  a  baseline  ordnance  item  are  considered  to 
determine  the  common  detections  for  the  ME  pair.  Thus,  the  distance  between  declara¬ 
tions  can  be  as  great  as  2.0  m.  The  different  fit  parameters  are  considered  to  establish 
trends  that  might  be  exploitable  for  combining  or  fusing  the  two  data  sets. 

3.4.1.1  Detection  Location  Offset 

The  location  offset  is  the  difference  between  the  fitted  location  of  each  detected 
baseline  item  (on  the  x-y  plane)  for  the  EMI  system  and  the  magnetometer.  To  determine 
any  positional  trends  in  the  actual  fitted  location  from  the  two  sensors,  the  relative  east 
and  north  position  of  fitted  location  for  the  sensor  pair  is  also  examined. 

Forty  baseline  items  are  common  detections  for  the  ME  sensor  pair.  The  mean 
radial  distance  between  detections  is  0.27  m,  with  a  standard  deviation  of  0.22  m. 
Twenty-four  common  declarations  have  a  location  offset  of  less  than  0.25  m  [see 
Figure  3-24(a)],  and  only  6  conunon  detections  are  separated  by  greater  than  0.5  m.  None 
of  the  common  detections  are  separated  by  more  than  0.88  m.  Figure  3-24(c)  shows  the 
location  error  for  the  magnetometer  versus  that  of  the  EMI  system.  It  is  evident  from  the 
figure  that  the  location  errors  are  highly  clustered  within  an  area  of  radius  0.5  m 
surrounding  the  baseline  item.  There  is  also  a  suggestion  that  large  location  errors  for  the 
EMI  system  tend  to  be  correlated  with  large  location  errors  for  the  magnetometer.  But  the 
corollary  is  not  true.  There  are  no  statistically  significant  easting  or  northing  location 
offsets  ([Figure  3-24(b)].  Both  the  mean  and  the  median  are  within  the  0.05-m  location 
error  estimate  for  the  RTK-DGPS. 

The  overall  performance  suggests  that  the  location  errors  exhibited  by  the  sensors 
are  small  and  tend  to  be  similar,  although  outliers  were  not  correlated.  The  location  error 
appears  to  be  caused  by  survey-to-survey  location  inaccuracies  and  noise  and  not  by  a 
statistically  significant  intrinsic  property  of  the  ordnance,  the  MTADS  system,  or  the 
MTR. 
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Figure  3-24.  Fitted  Location  Comparison  for  EMI  and  Magnetometer,  (a)  Distribution 
of  Distance  Between  EMi  and  Magnetometer  Location,  (b)  Difference  in  East 
versus  North,  (c)  Radial  Location  Error  for  Magnetometer  versus  EMI. 


3.4.1.2  Detection  Depth  Offset 

The  depth  offset  is  the  difference  between  the  fitted  depth  of  each  detected 
baseline  item  (on  the  x-y  plane)  for  the  two  sensors,  that  is,  the  target  depth  estimated 
from  the  EMI  data  subtracted  from  that  estimated  from  the  magnetometer  data.  The  depth 
offset  can  be  positive  or  negative.  The  trend  of  the  sign  indicates  whether  one  sensor 
consistently  estimates  the  depth  of  the  baseline  item  to  be  greater  than  the  fitted  depth 
from  the  other  sensor. 

In  the  ME  sensor  set,  33  of  40  baseline  items  have  a  depth  offset  within  +0.5  m  of 
one  another,  and  16  of  these  are  within  ±0.25  m.  There  were  four  detections  with  depth 
offsets  between  -1.0  and  -2.0  m  [see  Figure  3-25(a)];  these  are  due  to  the  relatively  poor 
depth  predictions  from  the  EMI  sensor  data  (and  considering  the  excellent  location 
accuracy,  probably  due  to  an  ineffective  model  for  calculating  depth).  The  general  trend 
is  that  the  EMI  analysis  produces  a  depth  that  is  shallower  than  that  estimated  from  the 
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Figure  3*25.  Fitted  Depth  Comparison  for  EMI  and  Magnetometer,  (a)  Distribution  of 
Depth  for  EMI  and  Magnetometer,  (b)  Depth  for  Magnetometer  versus  EMI. 

(c)  Radial  Distance  versus  Depth  Distance. 

magnetometer  data.  The  mean  depth  offset  for  the  pair  is  -0.33  m  (the  negative  value 
results  from  larger  depth  for  the  estimates  using  the  magnetometer  data)  with  a  standard 
deviation  of  0.46  m.  Thirty-three  out  of  forty  depth  estimations  for  the  EMI  are  less  than 
those  of  the  magnetometer. 

Figure  3-25(b)  shows  the  depth  error  for  the  EMI  sensor  data  relative  to 
magnetometer  sensor  data.  The  EMI  mean  depth  error  for  commonly  detected  targets  is 
-0.28  m  with  a  standard  deviation  of  0.39  cm,  while  the  magnetometer  has  a  mean  depth 
error  of  0.04  m  with  a  standard  deviation  of  0.23  m.  There  is  no  apparent  trend  between 
large  depth  error  in  one  sensor  and  large  depth  error  in  the  other. 

Figure  3-25(c)  shows  the  radial  distance  between  the  two  detections  and  the  depth 
distance  between  detections.  There  is  no  apparent  correlation  between  large  differences  in 
depth  and  larger  radial  differences. 
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3.4.1.3  Detection  GoF  Comparison 

Each  time  the  MTADS-DAS  attempts  to  fit  a  designated  anomaly  with  the  appro¬ 
priate  model,  a  measure  of  the  relative  fit  of  the  model  to  the  actual  data  is  established. 
Because  the  fit  methodology  requires  the  use  of  data  where  any  field  anomaly  caused  by 
the  presence  of  a  magnetic,  conducting,  or  magnetic  and  conducting  buried  anomaly  is 
less  than  the  background  level,  the  dynamic  range  of  the  GoF  parameter  is  small.  Thus,  a 
fit  of  less  than  0.90  is  quite  poor.  These  poor  fits  are  generally  associated  with  very  noisy 
background  or  atypical  compact  objects  in  the  ground.  Because  of  the  lack  of  robustness 
of  this  metric,  the  general  statistical  trends  resulting  from  a  comparison  using  the  ME  pair 
may  yield  very  little  insight  to  the  system  performance. 

Common  detections  for  both  the  magnetometer  and  EMI  system  exhibit  weakly 
correlated  GoFs  [see  Figure  3-26(a)].  Only  13  common  detections  have  both  fits 
>  90  percent  (this  is  the  entire  set  of  EMI  fits  at  that  level),  while  32  common  detections 
have  at  least  one  GoF  parameter  >  90  percent.  The  conclusion  from  the  general  trends  is 
that  the  sources  of  poor  GoFs  are  dependent  on  the  type  of  sensor.  There  appears  to  be 
little  additional  information  available  to  strengthen  the  detection/discrimination  process 
by  comparison  of  GoFs. 


Figure  3-26.  Comparison  of  Fitted  Parameters  for  Magnetometer  and  EMI. 
(a)  Magnetometer  GoF  versus  EMi  GoF.  (b)  Ferrous  Size  for 
Magnetometer  versus  EMI  Ferrous  Size. 


3.4.1.4  Detection  Size  Comparison 

The  MTADS-DAS  produces  several  other  fit  parameters.  The  set  related  to  size 
and  magnetic  moment  is  not  independent.  The  size  of  each  object  is  defined  as  the  effecc- 
tive  radius  of  a  sphere  with  a  designated  permeability  that  can  exhibit  a  geomagnetic  ally 
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induced  magnetic  moment  consistent  with  the  fitted  moment.  For  the  EMI  system,  which 
can  detect  both  ferrous  and  nonferrous  materials,  an  equivalent  radius  is  calculated  for 
each  material  type.  Here,  the  only  difference  between  the  calculations  is  magnetic  perme¬ 
ability  and  electrical  conductivity.  Thus,  the  ferrous  and  nonferrous  sizes  are  related  by  a 
simple  scaling  constant  (dependent  on  the  material  characteristics  chosen). 

The  ferrous  sizes  of  the  magnetometer  and  EMI  combination  are  correlated 
(correlation  coefficient  =  0.76)  [see  Figure  3-26(b)].  The  fitted  ferrous  size  of  all  but 
three  magnetometer  detections  was  larger  than  the  corresponding  fitted  ferrous  size  for 
the  EMI  sensor. 

3.4.2  False  Alarms  Common  to  the  ME  Pair 

To  determine  whether  there  are  trends  in  false-alarm  space,  we  analyzed  the  false 
alarms  eommon  to  the  ME  sensor  pair.  In  this  case,  determining  common  false  alarms  is 
nontrivial  because  contrary  to  the  common  detection  sets,  the  source  and  location  of  the 
false  alarm  is  unknown.  Thus,  we  used  the  following  process  to  establish  the  common 
false  alarms.  The  full  declaration  set  for  each  sensor  was  parsed  into  two  subsets:  declara¬ 
tions  that  match  baseline  items  and  declarations  that  are  false  alarms.  The  two  false-alarm 
sets  were  then  compared  by  location  only  (in  the  x-y  plane).  A  false  alarm  from  one  set 
within  1.0  m  (the  critical  false-alarm  radius)  of  a  false  alarm  from  another  set  was 
classified  as  a  common  false  alarm  for  that  sensor  pair.  It  is  possible  to  have  more  than 
one  false  alarm  from  one  sensor  set  within  the  designated  critical  false-alarm  radius  of  the 
second  set,  but  only  the  nearest  neighbor  pair  was  considered  in  this  analysis.  We  chose 
the  1.0-m  critical  false-alarm  radius  because  of  the  overwhelming  trend  to  common 
detection  of  the  baseline  items  of  less  than  0.5-m  location  offset;  1.0  m  is  approximately 
three  standard  deviations  from  the  mean  separation  for  a  detection  pair. 

Using  this  set  of  common  false  alarms,  we  examined  the  sensor  pair  for  trends  in 
location  offset,  depth  offset,  GoF,  and  size.  The  trends  in  the  data  show  some  degree  of 
correlation  between  the  fitted  parameters,  which  confirms  our  hypothesis  that  the  vast 
majority  of  false  alarms  are  the  result  of  compact,  near-surface  anomalies  that  act  as 
clutter  for  the  sensor. 

3.4.2.1  False-Alarms  Location  Offset 

From  the  full  set  of  197  EMI  and  608  magnetometer  false  alarms,  there  are  67,  86, 
92,  and  113  common  false  alarms  for  critical  false-alarm  radii  of  0.5,  0.75,  1.0,  and 
2.0  m,  respectively.  Figure  3-27(a)  shows  the  distribution  of  distance  between  the  false 
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alarm  locations  for  the  two  sensors.  The  distribution  has  a  median  at  about  0.38  m.  The 
long  tail  is  probably  the  result  of  labeling  two  discrete  false  alarms  a  common  false  alarm 
caused  by  a  single  clutter  item.  There  is  the  potential  that  false  alarms  that  are  not  really 
common  could  be  identified  by  other  fit  parameters,  such  as  depth.  Unluckily,  since  most 
clutter  is  within  the  first  half  meter  of  the  ground  (see  Figure  3-11),  false  alarms  from  two 
different  sources  are  likely  to  have  similar  depths. 


Figure  3-27.  Fitted  Location  Comparison  for  EMi  and  Magnetometer  Common  False 

Alarms,  (a)  Distribution  of  Distance  Between  EMI  and  Magnetometer  Location. 

(b)  Difference  in  East  versus  North. 

For  this  analysis,  common  false  alarms  are  considered  to  be  only  those  that  have  a 
radial  location  offset  of  less  than  1.0  m;  this  subset  of  false  alarms  comprises  roughly 
81  percent  of  the  common  false  alarms  at  a  critical  radius  of  2.0  m.  The  1.0-m  false-alarm 
critical  radius  is  based  on  the  three-standard-deviation  point  for  common  detections  for 
the  magnetometer  and  EMI  system. 

Figure  3-27(b)  shows  the  location  offset  for  the  sensor  pair  in  the  east  versus 
north  directions.  As  with  the  ordnance,  the  mean  and  median  offsets  in  each  direction  are 
less  than  the  5-cm  location  error  expected  from  the  RTK-DGPS.  In  addition,  both 
standard  deviations  are  approximately  30  cm. 
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3.4.2.2  False- Alarm  Depth  Offset 


For  the  ME  sensor  combination,  the  distribution  of  depth  offsets  reveals  that  7 1  of 
the  92  common  false  alarms  have  a  depth  offset  between  ±0.25  m.  Figure  3-28(a)  shows 
the  distribution  of  the  depth  offset.  Only  two  outliers  exist  in  this  plot:  one  where  the 
magnetometer  fits  the  anomaly  2.6  m  deeper  than  the  EMI  system  (this  is  probably  not  a 
common  detection)  and  one  where  the  EMI  system  fits  the  depth  approximately  1  m 
deeper  than  the  magnetometer.  The  EMI  sensor  appears  to  consistently  underestimate  the 
depth  of  the  target;  72  out  of  92  (78  percent)  common  false  alarms  have  the  EMI  detec¬ 
tion  depth  less  than  the  magnetic  detection  depth.  This  is  almost  identical  to  the  relative 
depth  for  the  commonly  detected  ordnance  (83  percent).  Thus,  there  is  no  apparent 
difference  between  the  depth  offset  for  ordnance  and  clutter. 


(a) 


-3  -2.5  -2  -1.5  -1  -0.5  0  0.5  1  1.5 

Distance  between  Depths  (m) 


Radial  Distance  between 
False  Alarms  (m) 

Figure  3-28.  Fitted  Depth  Comparison  for  EMI  and  Magnetometer  Common  False 
Alarms,  (a)  Distribution  of  Depth  for  EMI  and  Magnetometer. 

(b)  Radial  Distance  versus  Depth  Distance. 

Figure  3-28(b)  shows  a  scatter  plot  of  the  radial  location  offset  versus  the  depth 
offset  for  common  detections.  The  radial  offset  and  depth  offset  appear  to  be 
uncorrelated. 
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3.4.23  False- Alarm  GoF  Comparison 


The  correlation  between  GoFs  of  the  ME  pairs  is  very  low  [see  Figure  3-29(a)], 
perhaps  due  to  poor  fits  from  the  EMI  data.  As  the  figure  shows,  many  declarations  with 
a  GoF  over  0.9  for  the  magnetometer  have  corresponding  EMI  GoF  of  less  than  0.50  (22 
out  of  45).  This  contrasts  with  the  results  for  the  common  ordnance  detected.  There,  of 
the  30  ordnance  items  with  GoF  >0.9  for  the  magnetometer,  only  six  (20  percent)  have 
an  EMI  GoF  less  than  0.5.  This  suggests  that  the  EMI  fitting  algorithm  has  a  harder  time 
fitting  clutter  objects,  which  are  likely  to  be  asymmetric.  At  the  same  time,  most  objects 
exhibit  a  strong  magnetic  dipole  moment  that  fits  well  to  the  simple  model  for  magnetic 
signatures.  Thus,  a  difference  in  GoFs  for  the  two  systems  might  provide  potential  for 
discrimination  using  the  “fused”  data  set.  It  should  be  noted  that  the  reciprocal  pattern  is 
not  true.  Good  GoF  for  the  EMI  always  corresponds  to  good  GoF  for  the  magnetometer. 


EMI  Ferrous  Size  (m) 


Figure  3-29.  Comparison  of  Fitted  Parameters  for  Magnetometer  and  EMI  for  Common 
False  Alarms,  (a)  Magnetometer  GoF  versus  EMI  GoF.  (b)  Ferrous  Size  for 
EMI  versus  Magnetometer  Ferrous  Size,  (c)  Ferrous  Size  for 
EMI  versus  Magnetometer  Ferrous  Size  for  Small  Sizes. 


3-40 


3A.2.4  False-Alarm  Size  Comparison 

The  ferrous  sizes  of  the  magnetometer  and  EMI  combination  are  well  correlated 
(correlation  coefficient  =  0.84).  The  fitted  ferrous  size  of  all  but  seven  of  the  magnetom¬ 
eter  false  alarms  was  greater  than  or  equal  to  the  corresponding  fitted  ferrous  size  for  the 
EMI  sensor.  Figure  3-29(b)  shows  a  scatter  plot  of  fitted  ferrous  size  for  the  magnetom¬ 
eter  and  EMI.  Figure  3-29(c)  is  a  scatter  plot  of  the  smallest  sized  objects. 
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4.0  NRL  DECLARATION  ANALYSIS 


This  chapter  contains  an  analysis  of  target  declarations  selected  by  NRL  using  the 
MTADS-DAS  for  the  three  surveys'  at  the  MTR  at  Twentynine  Palms.  NRL  did  all 
analysis  and  provided  IDA  three  spread  sheets  (one  each  for  magnetometer,  magnetic 
gradiometer,  and  EMI)  containing  the  analysis  results.  As  with  the  analysis  of  the  IDA 
declaration  presented  in  Chapter  3,  NRL  examined  deteetion  and  false-alarm 
performance  and  analyzed  sensor-specific  declaration  parameters  provided  by  the 
MTADS-DAS.  The  data  from  the  MTADS-DAS  include  several  parameters  used  to 
characterize  sensor  performance.  For  the  magnetometer  and  magnetic  gradiometer,  these 
parameters  include  magnitude  of  the  fit  magnetic  moment,  inclination  and  declination  of 
the  fit  magnetic  moment  (and  also  ferrous  size),  depth,  northing  coordinate,  easting 
coordinate,  and  GoF.  For  the  EMI  sensor  the  parameters  include  depth,  northing 
coordinate,  easting  coordinate,  ferrous  and  nonferrous  size,  and  GoF.  Along  with 
detection  performance  (this  includes  both  detection  rates  and  false-alarm  rates),  the  fitted 
parameters  for  each  detection  are  analyzed.  In  addition,  trends  and  common  features  for 
both  detections  and  false  alarms  are  presented  for  the  ME  sensor  pair. 

The  chapter  is  broken  down  into  six  sections.  The  first  section  presents  the 
detection  and  false-alarm  rates  and  a  discussion  of  false-alarm  mitigation  applied  to  the 
NRL  magnetometer  and  magnetic  gradiometer  data.  The  seeond  seetion  is  an  analysis  of 
the  location  and  depth  accuracy  relative  to  fitted  parameters  available  from  the  MTADS- 
DAS.  The  third  section  gives  an  evaluation  of  the  trends  in  both  detection  and  false-alarm 
space  for  the  ME  sensor  pair.  The  fourth  section  is  a  comparison  of  the  results  obtained 
by  IDA  (presented  in  Chapter  3)  and  those  obtained  by  NRL  during  its  sensor  data  analy¬ 
sis.  The  final  two  sections  present  information  on  unique  false  alarms  and  data  fusion 
using  logical  operations. 


'  These  three  survey  runs  are  the  same  runs  analyzed  by  IDA  in  Chapter  3.  For  that  reason  it  is  possible 
to  do  a  one-to-one  comparison  between  the  IDA  results  and  the  NRL  results. 
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4.1  GENERAL  PERFORMANCE  RESULTS 


We  compare  each  declaration  data  set  generated  by  NRL  using  the  MTADS-DAS 
to  the  emplaced  target  baseline,  using  only  the  nongrouped  ordnance  (see  Section  3.1.1). 
To  match  target  declarations  to  the  baseline,  we  determine  if  the  relative  northing/easting 
position  of  the  declaration  is  within  the  critical  radius,  of  the  center  of  the  emplaced 
ordnance  item.  If  the  declaration  falls  outside  the  declaration  is  classified  as  a  false 
alarm.  Because  of  NRL’s  substantial  field  experience  with  the  MTR  at  Twentynine 
Palms,  their  performance  may  be  enhanced  by  man-in-the-loop  discrimination. 

4.1.1  Detection  and  False- Alarm  Rates 

A  modified  declaration  set  is  produced  for  each  sensor  type.  For  the  magnetom¬ 
eter  and  magnetic  gradiometer,  only  a  relative  inclination  test  is  used  to  discriminate 
between  potential  targets  and  potential  false  alarms.  This  abbreviated  discrimination 
algorithm  is  used  because  the  relative  inclination  discriminator  is  more  robust  than  the 
other  contributions  used  on  the  IDA  declaration  set,  and  the  experienced  operator  visually 
preprocessed  the  data  to  eliminate  targets  that  were  too  small  or  on  the  surface.  Thus, 
only  relative  inclination  is  needed  to  perform  a  second  round  of  discrimination.  The 
conditions  used  are  the  following; 

•  Confidence  level  0 — The  relative  inclination  is  greater  than  40  deg. 

•  Confidence  level  1 — The  relative  inclination  is  between  35  deg  and  40  deg 
and  all  of  confidence  level  0. 

•  Confidence  level  2 — The  relative  inclination  is  between  30  deg  and  35  deg 
and  all  of  confidence  level  1 . 

Detection  probabilities  and  false-alarm  rates  can  be  determined  for  the  three  sets 
of  data.  Table  4-1  lists  Pj,  FAR,  and  Pfa  for  an  of  0.5  m  and  1.0  m.  Here,  as  in 
Chapter  3,  after  use  of  the  discrimination  algorithm,  is  represented  by  a  surrogate:  the 
fraction  of  the  test  site  covered  by  false-alarm  areas  with  radius.  The  FAR  is  calcu¬ 
lated  assuming  an  approximately  8-acre  test  site  (30,686.6  m^).  No  attempt  has  been 
made  to  account  for  areas  not  covered  by  MTADS. 

The  ROC  approach  provides  a  mechanism  to  evaluate  the  relative  performance  of 
systems  or  algorithms  for  a  given  Figures  4-1  and  4-2  show  the  probability  of  detec¬ 
tion  versus  probability  of  false  alarm  for  all  sensors  at  of  0.5  m  and  1.0  m,  respec¬ 
tively.  The  solid  lines  are  isoperformance  curves  (discussed  in  Chapter  3)  calculated  by 
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Table  4-1.  Detection  and  False-Alarm  Performance  of  MTADS  irom  NRL  Analysis 


Rent  0.5  m 

Rent  ^.Om 

Sensor^ 

P, 

FAR 

(m-^) 

Pfa 

P, 

FAR 

(m-®) 

P,a 

Magnetometer 
(Full  Data  Set) 

0.694 

7.7x10-® 

6.1  X  10-® 

0.823 

7.4x  10-® 

2.4x10"® 

Magnetometer 
(Confidence  Level  2) 

0.677 

5.1  X  10-® 

4.1  X  10-® 

0.790 

4.9x  10-® 

1.5  X  10“® 

Magnetometer 
(Confidence  Level  1) 

0.661 

4.9x10“® 

3.9  X  10“® 

0.774 

4.7  X  10"® 

1.5x10“® 

Magnetometer 
(Confidence  Level  0) 

0.645 

4.6  X  10"® 

3.6  X  10  ® 

0.742 

4.4x10-® 

1.4x10"® 

Magnetic  Gradiometer 
(Full  Data  Set) 

0.548 

8.2  X  10"® 

6.5  X  10  ® 

0.774 

7.7x  10-® 

2.4  X  10"® 

Magnetic  Gradiometer 
(Confidence  Level  2) 

0.516 

5.4x  10“® 

4.2  X  10  ® 

0.726 

4.9x10"® 

1.6x10“® 

Magnetic  Gradiometer 
(Confidence  Level  1) 

0.516 

4.9  X  10"® 

3.9  X  10  ® 

0.726 

4.4  X  10"® 

1.4  X  10"® 

Magnetic  Gradiometer 
(Confidence  Level  0) 

0.500 

4.6x10-® 

3.6  X  10  ® 

0.710 

4.1  X  10"® 

1.3x10"® 

Electromagnetic 

Induction 

0.790 

4.9x  10“® 

3.8  X  10  ® 

0.839 

4.8x  10“® 

1.5x  10“® 

*  The  declaration  set  determined  by  using  the  MTADS-DAS  is  considered  the  full  data  set.  For  this  set 
only  magnetic  moment  modeling  was  used  to  reduce  the  number  of  declarations.  Confidence  Level  2 
refers  to  the  set  of  declarations  with  a  final  IDA  ranking  of  2  or  less  (relative  inclination  greater  than 
30  deg).  Confidence  Level  1  refers  to  the  set  of  declarations  with  a  final  IDA  ranking  of  1  or  less  (relative 
inclination  greater  than  35  deg).  Confidence  Level  0  refers  to  the  set  of  declarations  with  a  final  IDA 
ranking  of  0  (relative  inclination  greater  than  40  deg). 

using  a  Gaussian  model  for  the  distribution  functions  of  the  target  and  clutter/noise 
signatures.^  The  multiple  points  presented  for  each  sensor  are  related  to  the  false-alarm 
discrimination  algorithm  using  the  relative  inclination.  Looking  at  an  R^^i,  of  0.5  m 
(Figure  4-1)  for  the  four  points  (representing  confidence  levels  0,  1,2,  and  the  full  set) 
that  result  from  implementation  of  the  discrimination  algorithm,  the  magnetometer 
performance  does  not  improve  over  the  three  different  confidence  levels  (the  three  points 
fall  on  a  single  isoperformance  curve  within  statistical  error).  There  is,  however,  a  slight 
performance  improvement  using  the  discrimination  algorithm  over  that  of  the  full  magne¬ 
tometer  data  set.  The  trend  for  an  of  1 .0  m  is  a  little  different.  There  is  virtually  no 
performance  improvement  from  the  full  magnetometer  set  to  the  confidence  levels  1  and 
2  sets,  and  a  decrease  in  performance  at  confidence  level  0.  Note  that  the  statistical 
significance  of  this  is  limited  because  of  the  small  number  of  targets. 


^  The  Gaussian  model  used  here  assumes  the  same  variance  for  the  noise/clutter  and  the  ordnance. 
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Pfa 


Figure  4-1.  Probability  of  Detection  versus  Probability  of  False  Alarm  for  the  Magnetometer 
(Diamonds),  Magnetic  Gradiometer  (Circles),  and  EMI  System  (Triangle).  Rcri,  is  0.5  m.  For 
the  magnetometer  and  magnetic  gradiometer,  confidence  levels  of  0, 1,  and  2  (using  only 
relative  inclination)  and  full  data  set  are  circles  from  left  to  right,  respectively. 


Figure  4-2.  Probability  of  Detection  versus  Probability  of  False  Alarm  for  the  Magnetometer 
(Diamonds),  Magnetic  Gradiometer  (Circles),  and  EMI  System  (Triangle).  Rent  is  1.0  m.  For 
the  magnetometer  and  magnetic  gradiometer,  confidence  levels  of  0, 1,  and  2  (using  only 
relative  inclination)  and  full  data  set  are  circles  from  left  to  right,  respectively. 
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Figures  4-1  and  4-2  also  show  performance  of  the  magnetic  gradiometer  using  the 
ROC  approach  for  of  0.5  m  and  1.0  m,  respectively.  For  the  magnetic  gradiometer, 
the  slightly  improved  performance  seen  for  the  magnetometer  is  again  seen  for  the  0.5-m 
The  performance  remains  the  same  with  implementation  of  the  discrimination 
algorithm  for  of  1 .0  m. 

After  implementation  of  the  relative  inclination  discrimination  algorithm,  the  EMI 
sensor  has  the  best  overall  performance,  exceeding  even  the  magnetometer  and  magnetic 
gradiometer  performance.  The  performance  of  the  magnetometer  is  better  than  that  of  the 
magnetic  gradiometer,  even  for  the  full  data  set. 

4.1.2  Performance  Dependence  on  Critical  Radius 

As  discussed  above,  the  probability  of  detection  and  false-alarm  rate  are  depend¬ 
ent  on  the  used  to  determine  detections.  Figures  4-3(a)-(c)  show  the  probability  of 
detection  and  number  of  detections  as  a  function  of  for  the  magnetometer,  magnetic 
gradiometer,  and  EMI  system,  respectively.  For  the  magnetic  detection  systems,  the 
number  of  detections  increases  slowly  above  an  of  approximately  1  m.  This  gradual 
increase  in  the  number  of  ordnance  items  detected  above  an  of  1.0  m  may  be  caused 
by  detections  that  are  “lucky”  matches  (Altshuler  et  al.,  1996)  or  by  potentially  true 
detections  with  very  poor  location  accuracy  (more  than  three  standard  deviations  above 
the  mean).  The  EMI  detection  system  exhibits  even  better  versus  performance, 
with  no  significant  increase  in  detection  rates  above  approximately  0.6  m.  Table  4-2 
contains  the  mean  and  standard  deviation  of  the  radial  location  accuracy  and  the  depth 
accuracy^  (both  true  and  absolute)  for  all  three  sensor  configurations  for  a  1-m  critical 
radius.  The  EMI  system  exhibits  the  best  location  accuracy,  but  also  shows  the  largest 
depth  error,  with  a  trend  of  underestimating  the  depth  of  the  ordnance  item.  In  all  cases, 
for  both  location  and  depth  accuracy,  the  MTADS  platform  and  MTADS-DAS  performed 
quite  well. 

4.1.3  False-Alarm  Mitigation  Technique 

A  relative  inclination-based  false-alarm  mitigation  technique  is  used  to  sort  poten¬ 
tial  ordnance  from  nonordnance  for  both  the  magnetometer  and  magnetic  gradiometer 


3  Both  accuracy  and  error  are  used  in  this  document  to  refer  to  the  difference  in  the  location  and  depth 
determined  by  the  MTADS-DhS  fitting  algorithm  and  the  baseline  emplaced  position  of  ordnance. 
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(a) 


Figure  4-3.  Detection  Performance  versus  Criticai  Radius  for  the  Emplaced  Targets, 
(a)  Magnetometer,  (b)  Magnetic  Gradiometer.  (c)  EMI  System. 


Table  4-2.  Detection  Location  Error  and  Depth  Error  for  1-m  Critical  Radius 


Radial 

Depth 

Magnitude  Depth 

Sensor 

Mean 

Standard 

Mean 

Standard 

Mean 

Standard 

Error 

Deviation 

Error 

Deviation 

Error 

Deviation 

(m) 

(m) 

(m) 

(m) 

(m) 

(m) 

Magnetometer 

0.28 

0.24 

0.10 

0.39 

0.26 

0.31 

Magnetic 

Gradiometer 

0.37 

0.26 

0.09 

0.66 

0.28 

0.61 

EMI 

0.27 

0.16 

-0.19 

0.52 

0.37 

0.41 

declaration  set.  The  false-alarm  mitigation  is  performed  using  a  confidence  rankings 
scheme  that  weights  each  declaration  based  on  the  orientation  of  the  magnetic  moment 
relative  to  the  geomagnetic  field.  The  false-alarm  mitigation  technique  relies  on  a 
preferential  direction  of  the  ordnance  magnetic  moment.  This  technique  assumes 
ordnance  magnetization  is  dominated  by  that  magnetization  induced  by  the  geomagnetic 
field;  i.e.,  the  remanent  magnetization  is  small.  On  the  other  hand,  the  source  of  false 
alarms,  clutter,  is  predicted  to  exhibit  a  more  uniform  angular  distribution  of  magnetic 
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moments.  Figures  4-4(a)  and  4-5(a)  show  the  distribution  of  magnetic  moment  direction 
for  the  detected  ordnance  for  the  magnetometer  and  magnetic  gradiometer,  respectively. 
The  cut-off  for  the  algorithm  used  for  the  NRL  data  to  separate  moments  with  a 
confidence  level  of  2  or  less  is  approximately  30  deg.'^  The  magnetometer  data  show  only 
2  out  of  5 1  ordnance  items  with  a  relative  inclination  of  less  than  30  deg.  In  this  figure 
the  data  is  binned  in  5-deg  groups  centered  at  5-deg  intervals,  i.e.,  25,  30,  35,  etc.  The 
magnetic  gradiometer  produces  similar  number  of  ordnance  items  that  possess  a  relative 
inclination  below  30  deg  (3  out  of  48). 

Figures  4-4(b)  and  4-5(b)  show  the  distribution  of  the  relative  inclination  of  the 
false  alarms.  For  the  magnetometer  there  is  a  significant  population  of  the  angular 
distribution  for  nonordnance  with  a  relative  inclination  below  30  deg,  in  contrast  to  the 
ordnance  population.  In  addition,  the  distribution  function  for  false  alarms  from  the 
magnetic  gradiometer  also  exhibits  a  significant  population  below  a  relative  inclination 
30  deg. 

4.1.4  False  Alarms 

The  depth  distribution  for  false  alcums  exhibits  an  interesting  difference  between 
the  active  EMI  system  and  the  passive  magnetostatic  detection  techniques.  Figure  4-6 
shows  the  number  of  false  alarms  as  a  function  of  depth  for  each  sensor.  Although  for  all 
sensors  there  are  a  substantial  number  of  false  alarms  in  the  first  0.1  m  of  the  ground,  the 
EMI  does  not  produce  many  additional  false  alarms  at  greater  depths  (only  3  out  146 
false  alarms  exist  below  1  m).  On  the  other  hand,  for  the  magnetometer  and  magnetic 
gradiometer,  the  distribution  has  approximately  10  percent  of  the  false  alarm  population 
at  depths  below  1  m.^  This  feature  may  provide  some  method  for  discrimination,  but  a 
number  of  issues  must  be  addressed. 

•  The  EMI  system  shows  poor  depth  accuracy  (larger  than  the  bins  used  to 
generate  Figure  4-6).  Thus,  the  clutter  responsible  for  the  EMI  anomaly  may 
have  a  different  depth  distribution  function  than  shown  in  Figure  4-6. 

•  The  magnetometer  and  magnetic  gradiometer  tend  to  perform  more  poorly 
detecting  small  objects  near  the  surface  (these  can  be  UXO)  than  the  EMI 
system. 


^  The  angle  used  is  measured  with  the  geo-coordinate  rotated  so  that  the  geomagnetic  field  is  along  the 
z-axis.  This  results  in  a  geomagnetic  field  inclination  of  90  deg.  Thus,  when  this  report  refers  to 
relative  inclination,  the  reference  is  the  rotated  geo-coordinate  system. 

^  In  this  analysis  the  data  show  a  greater  percentage  of  false  alarms  at  greater  depths  compared  to  the 
targets  picked  by  IDA  discussed  in  Section  3.1.4.  This  difference  is  the  result  of  the  depth 
discrimination  done  by  the  analyst  at  NRL  responsible  for  picking  targets. 
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Figure  4-4.  Distribution  of  Magnetic  Moments  Relative  to  the  Geomagnetic  Field  as 
Measured  by  the  Magnetometer.  Relative  Inclination  of  90  deg  is  Aligned  with  the 
Geomagnetic  Field.  Relative  Inclination  of  0  deg  is  Orthogonal  to  Geomagnetic  Field. 

(a)  Ordnance,  (b  )  False  Alarms. 


Relative  Inclination  (deg.) 


Figure  4-5.  Distribution  of  Magnetic  Moments  Relative  to  the  Geomagnetic  Field  as 
Measured  by  the  Magnetic  Gradiometer.  Relative  Inclination  of  90  deg  is  Aligned 
with  the  Geomagnetic  Field.  Relative  Inclination  of  0  deg  is  Orthogonal  to 
Geomagnetic  Field,  (a)  Ordnance,  (b )  False  Alarms. 
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Fitted  Depth  (m) 


Figure  4-6.  Number  of  False  Alarms  as  a  Function  of  Fitted  Depth, 
(a)  Magnetometer,  (b)  Magnetic  Gradiometer.  (c)  EMI  System. 


4.2  SENSOR  PERFORMANCE 

This  section  characterizes  the  system  location  performance  for  detected  ordnance 
by  evaluating  the  MTADS-DAS  parameters  and  determining  trends.  For  this  set  of  calcu¬ 
lations,  the  grouping  of  targets  is  not  used.  For  each  sensor  type,  the  entire  declaration  set 
is  matched  to  ungrouped  sets  of  emplaced  ordnance.  A  critical  radius  of  1  m  is  used  to 
determine  the  set  of  declarations  that  match  emplaced  ordnance  from  the  target  baseline. 
Within  a  1-m  critical  radius,  analysis  of  the  NRL  declaration  set  resulted  in  the  following 
detection  statistics: 

•  51  out  of  62  (82.3  percent)  ungrouped  baseline  items  with  the  magnetometer, 

•  48  out  of  62  (77.4  percent)  baseline  items  with  the  magnetic  gradiometer,  and 

•  52  out  of  62  (83.9  percent)  baseline  items  with  the  EMI  sensor. 

Below  we  discuss  physical  trends  for  each  group  of  baseline  items  detected  by  a  sensor. 
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4.2.1  Sensor  Performance:  Radial  Location  Accuracy 

For  each  sensor  declaration  which  matches  an  emplaced  ordnance  item  within  the 
1-m  critical  radius,  the  coordinates  of  the  fitted  anomaly  and  radial  location  accuracy  or 
radial  location  error  are  recorded  (radial  location  accuracy  =  fitted  anomaly  location  - 
actual  location  of  detected  ordnance  item).  The  results  of  the  analysis  follow. 

4.2.1.1  Magnetometer 

The  mean  radial  location  accuracy  for  the  targets  detected  by  the  MTADS 
platform  is  0.28  m,  with  a  standard  deviation  of  0.24  m.  Since  the  fitted  dipole  moment  is 
weakly  correlated  to  the  location  accuracy,  the  magnitude  of  the  radial  location  error  does 
not  appear  to  affect  the  magnitude  of  the  fit  magnetic  moment  of  the  selected  anomaly 
[see  Figure  4-7(a)]. 


0  0.5  1  0  0.5  1 


Radial  Location  Error  (m) 

Figure  4-7.  Radial  Location  Error  for  the  Magnetometer,  (a)  Versus  Fit  Magnetic  Moment, 
(b)  Versus  GoF.  (c)  Versus  Depth,  (d)  Versus  Depth  Error. 

The  GoF  parameter  is  found  not  to  be  an  accurate  metric  for  the  radial  location 
accuracy  [see  Figure  4-7(b)].  There  is  a  general  trend  that  location  accuracy  decreases 
with  decreasing  GoF.  A  GoF  greater  than  0.85  is  a  strong  indication  of  a  low  radial  error: 
only  2  of  the  33  detected  baseline  items  with  a  GoF  greater  than  0.85  have  a  radial  error 
larger  than  0.5  m.  But  a  low  fit  parameter  does  not  correlate  with  the  radial  error. 
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Eighteen  detected  ordnance  items  have  a  GoF  of  0.85  or  below.  Of  those,  twelve  have  a 
radial  error  less  than  0.5  m.  In  addition,  four  detected  ordnance  items,  ranging  from  a 
60-mm  mortar  to  750-lb  bomb,  have  GoF  less  than  0.75  (see  Table  4-3).  The  radial  error 
for  these  four  ordnance  items  ranges  between  0.14  m  and  0.91  m. 


Table  4-3.  Radial  and  Depth  Errors  for  Poorly  Fit  Ordnance  Detected  by  Magnetometer 


Description 

GoF 

Radial  Error  (m) 

Relative  Depth  Error 

60  mm 

0.66 

0,28 

0.55 

250  lb 

0.69 

0.14 

-1.18 

500  lb 

0.66 

0.84 

0.25 

750  lb 

0.60 

0.91 

-0.05 

There  is  minimal  correlation  between  depth  or  depth  error  and  location  accuracy 
[see  Figure  4-7 (c)  and  (d)].  Again,  the  best  locator  accuracy  is  often  associated  with  good 
depth  accuracy,  but  the  trend  is  not  compelling. 


4.2.1. 2  Magnetic  Gradiometer 

The  mean  radial  location  accuracy  for  the  targets  detected  by  the  MTADS  plat¬ 
form  is  0.37  m  with  a  standard  deviation  of  0.24  m.  A  plot  of  the  magnitude  of  the  fit 
magnetic  moment  versus  radial  location  error  reveals  a  low  correlation  between  these  fit 
parameters  [see  Figure  4-8(a)]. 

GoF  for  the  magnetic  gradiometer  is  not  as  good  a  measure  as  it  is  for  the 
magnetometer  [see  Figure  4-8(b)].  There  are  1 1  detected  targets  with  GoF  less  than  0.5 
(23  percent),  2  of  which  had  radial  errors  less  than  0.5  m.  Almost  43  percent  have  a  GoF 
of  0.75  or  less  compared  to  less  than  10  percent  for  the  magnetometer,  and  11 
(23  percent)  have  a  GoF  less  than  0.5  (see  Table  4-4).  Deeper  objects  show  more  scatter 
in  location  accuracy  than  do  shallow  objects  [see  Figures  4-8(c)  and  (d)].  As  expected, 
the  fitted  target  depth  did  not  correlate  with  the  location  accuracy,  although  the  largest 
depth  errors  had  large  radial  location  errors. 

4.2.1.3  EMI 

The  mean  radial  location  accuracy  for  the  detected  baseline  items  is  0.22  m  with  a 
standard  deviation  of  0.16  m.  Comparing  target  selection  of  the  individual  sensors,  the 
EMI  system  has  the  best  location  accuracy.  The  size  of  the  object  does  not  seem  to  be 
well  correlated  with  the  location  accuracy  [Figure  4-9(a)]. 
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Figure  4-8.  Radial  Location  Error  for  the  Magnetic  Gradiometer.  (a)  Versus  Fit 
Magnetic  Moment,  (b)  Versus  GoF.  (c)  Versus  Depth,  (d)  Versus  Depth  Error. 

Table  4-4.  Radial  and  Depth  Errors  for  Poorly  Fit  Ordnance  Detected  by 


Magnetic  Gradiometer 

Baseline  item 

GoF 

Radial  Error  (m) 

Relative  Depth  Error 

8  in. 

0.01 

0.74 

0.88 

60  mm 

0.06 

0.41 

0.06 

155  mm 

0.08 

0.76 

-0.10 

81  mm 

0.25 

0.12 

-0.09 

105  mm 

0.31 

0.52 

-0.38 

250  lb 

0.33 

0.66 

0.02 

105  mm 

0.35 

0.63 

0.47 

105  mm 

0.40 

0.93 

-0.61 

60  mm 

0.43 

0.91 

-0.15 

60  mm 

0.48 

0.97 

-0.12 

250  lb 

0.49 

0.81 

0.66 

There  were  1 1  targets  with  a  GoF  less  than  0.40;  the  radial  location  accuracy  of 
this  subgroup  of  targets  was  less  than  0.7  m  [see  Figure  4-9(b)].  The  GoF  parameter  is 
not  a  good  metric  for  location  accuracy:  poor  correlation  is  exhibited.  In  addition,  the 
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fitted  depth  of  the  object  was  not  found  to  have  a  strong  relation  with  the  location  error. 
Figures  4-9(c)  and  (d)  show  that  there  is  more  scatter  in  depth  error  for  large  errors. 
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Figure  4-9.  Radial  Location  Error  for  the  EMI  System,  (a)  Versus  Nonferrous  Size, 
(b)  Versus  GoF.  (c)  Versus  Depth,  (d)  Versus  Depth  Error. 


4.2.2  Sensor  Performance:  NRL  Depth  Accuracy 

In  this  section,  we  analyze  the  NRL  depth  accuracy  for  the  targets  which  matched 
baseline  items  within  a  1-m  halo. 


4.2.2.1  Magnetometer 

The  average  depth  offset^  for  the  detected  baseline  items  is  0.26  m,  with  a 
standard  deviation  of  0.31  m.  The  mean  relative  depth  offset^  was  0.173,  with  a  standard 
deviation  of  0.200.  Table  4-5  lists  the  targets  having  relative  depth  errors  greater  than 
+0.25. 


^  Depth  offset  is  the  positive  difference  between  the  fit  depth  and  the  actual  depth  of  a  baseline  item. 

^  Relative  depth  offset  is  the  depth  offset  normalized  to  the  lit  depth  of  the  baseline  item.  Relative  depth 

error  maintains  the  sign. 
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Table  4-5.  Ordnance  with  Large  Relative  Depth  Error 


Description 

Rel.  Depth  Error 

Actual  Depth  (m) 

Radial  Error  (m) 

Fit 

250  lb 

-1.18 

2.38 

0.14 

0.69 

81  mm 

0.64 

0.61 

0.61 

0.62 

155  mm 

-0.57 

1.19 

0.10 

0.84 

60  mm 

0.56 

0.15 

0.05 

0.93 

60  mm 

0.55 

0.18 

0.28 

0.66 

60  mm 

0.49 

0.46 

0.29 

0.75 

105  mm 

0.48 

1.01 

0.58 

0.82 

500  lb 

-0.40 

1.22 

0.34 

0.98 

8  in. 

0.30 

2.07 

0.20 

0.75 

The  objects  with  large  relative  depth  errors  tend  to  be  objects  with  small  magnetic 
signatures  buried  within  1  m  of  the  surface.  The  clutter  may  have  contributed  to  the  poor 


GoF. 


There  is  a  weak  relation  between  the  fit  depth  and  the  relative  depth  error  (the 
linear  correlation  coefficient  is  -0.72):  the  near-surface  objects  tend  to  have  higher 
relative  depth  errors,  as  would  be  expected. 

There  is  poor  correlation  between  the  magnetic  moment  and  depth  error  [see 
Figure  4- 10(a)].  There  also  is  no  general  trend  between  magnetometer  moment  and 
relative  depth  error. 

GoF  is  a  weak  metric  for  depth  accuracy:  the  correlation  coefficient  between 
depth  and  GoF  is  close  to  zero  [see  Figure  4- 10(d)].  The  correlation  between  GoF  and 
relative  depth  error  is  also  very  nearly  zero,  which  suggests  that  the  relative  depth  errors 
can  not  be  attributed  to  poor  fits.  Very  deep  objects  (>3  m)  had  radial  errors  between 
0.6  and  1.0  m.  Six  out  of  ten  targets  with  fit  depth  greater  than  3  m  were  detected.  Among 
these  targets  were  a  500-lb  bomb,  a  750-lb  bomb,  and  an  8-in.  shell. 

4.2.2.2  Magnetic  Gradiometer 

The  average  depth  offset  for  the  baseline  items  detected  by  the  magnetic  gradiom¬ 
eter  is  0.28  m,  with  a  standard  deviation  of  0.61  m.  Excluding  one  8-in.  shell  with  an 
anomalous  depth  error  of  4  m,*  the  average  depth  offset  is  0.19  m,  with  a  standard 


^  IDA  also  found  this  8-in.  shell  with  a  depth  error  of  0.14  m  and  a  radial  error  of  0.64  m. 
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Figure  4-10.  Depth  and  Depth  Error  for  the  Magnetometer,  (a)  Depth  Error  versus  Fit 
Magnetic  Moment,  (b)  Depth  Error  versus  GoF.  (c)  Depth  Error  versus  Depth. 

(d)  Depth  versus  GoF. 


deviation  of  0.22  m.  The  average  relative  depth  error  is  0.20  m,  with  a  standard  deviation 
of  0.18  m.  There  were  11  targets  with  relative  depth  errors  greater  than  0.25:  one  500-lb 
bomb,  two  250-lb  bombs,  one  155-mm  shell,  three  105-mm  shells,  two  60-mm  mortcurs, 
and  two  8-in.  shells. 


The  magnitude  of  the  magnetic  moment  shows  some  correlation  with  the  depth 
error  [Figure  4-1 1(a)]:  we  found  the  correlation  coefficient  to  be  0.76.  The  8-in.  shell  that 
is  responsible  for  the  4-m  depth  error  also  has  the  largest  fitted  dipole  moment.  The 
magnitude  of  the  magnetic  moment  was  not  correlated  with  the  relative  depth  error. 

There  is  no  correlation  between  fit  depth  and  the  GoF.  There  is  also  virtually  no 
correlation  between  relative  depth  error  and  GoF;  of  the  11  targets  with  relative  depth 
error  greater  than  0.25,  five  targets  had  a  GoF  greater  than  80  percent.  There  was  no 
strong  trend  between  fit  depth  and  relative  depth  error;  therefore,  we  cannot  say  deeper 
objects  had  larger  depth  errors. 
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Figure  4-11.  Depth  and  Depth  Error  for  the  Magnetic  Gradiometer.  (a)  Depth  Error  versus 
Fit  Magnetic  Moment,  (b)  Depth  Error  versus  GoF.  (c)  Depth  Error  versus  Depth. 

(d)  Depth  versus  GoF. 


4.2.2.3  EMI 

The  average  fit  depth  error  is  -0.19  (an  underestimation  of  depth)  with  a  standard 
deviation  of  0.52  m.  Table  4-6  lists  the  eight  baseline  items  having  relative  depth  errors 
1 .00  or  larger. 


Tabie  4-6.  Ordnance  with  Reiative  Depth  Error  Greater  than  1.00 


Description 

Rel.  Depth  Error 

Actual  Depth  (m) 

Radial  Error  (m) 

GoF 

60  mm 

-45.00 

0.46 

0.26 

0.33 

105  mm 

-29.5 

0.61 

0.08 

0.72 

81  mm 

-7.71 

0.61 

0.12 

0.46 

105  mm 

-5.57 

0.46 

0.10 

0.90 

60  mm 

-2.07 

0.46 

0.22 

0.71 

81  mm 

-1.24 

0.76 

0.14 

0.53 

5001b 

-1.11 

1.92 

0.39 

0.90 

50  lb 

-1.00 

3.96 

0.65 

0.07 
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The  fit  depth  of  all  the  baseline  items  in  the  above  table  is  less  than  the  actual 
target  depth.  In  addition,  six  of  the  eight  targets  were  emplaced  between  0.46  m  and 
0.61  m. 


The  correlation  between  EMI  nonferrous  size  and  depth  error  is  low  (correlation 
coefficient  =  0.57)  [see  Figure  4- 12(a)],  suggesting  that  size  appears  to  have  a  weak 
effect  on  the  depth  error. 


With  a  linear  correlation  coefficient  of  -0.56,  the  correlation  of  GoF  with  depth  is 
also  quite  weak  [see  Figure  4- 12(b)].  Of  the  52  baseline  items  detected  by  the  EMI 
sensor,  13  targets  have  a  GoF  less  than  50  percent,  and  6  have  a  GoF  less  than  30  percent. 
It  is  therefore  not  reasonable  to  employ  GoF  as  the  sole  selection  parameter. 


Depth  Error  (m) 


Figure  4-12.  Depth  and  Depth  Error  for  the  EMI.  (a)  Depth  Error  versus 
Nonferrous  Size,  (b)  Depth  versus  GoF. 


4.3  DETECTIONS  AND  FALSE  ALARMS  COMMON  TO 
MAGNETOMETER  AND  EMI 

We  analyze  the  declarations  common  to  the  magnetometer  and  the  EMI  systems 
to  understand  the  general  trends  associated  with  detection  by  multiple  systems.  A 
common  detection  is  declared  for  all  ordnance  matched  to  a  declaration  of  both  sensors  in 
the  pair.  A  common  false  alarm  is  declared  for  any  false  alarms  within  1.0  m  of  each 
other. 


4.3.1  Common  Detections 

All  declarations  within  a  1 .0-m  of  ^  baseline  ordnance  item  are  considered  to 
determine  the  common  detections  for  the  ME  pair.  Thus,  the  distance  between 
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declarations  can  be  as  great  as  2.0  m.  The  different  fit  parameters  are  considered  to 
establish  trends  that  might  be  exploitable  for  combining  or  fusing  the  two  data  sets. 

4.3. 1.1  Detection  Location  Offset 

Forty-seven  baseline  items  are  common  detections  for  the  ME  sensor  pair. 
The  mean  radial  distance  between  detections  is  0.29  m,  with  a  standard  deviation  of 
0.21  m.  Twenty-four  common  declarations  had  a  location  offset  of  less  than  0.25  m  [see 
Figure  4- 13(a)],  and  only  7  common  detections  were  separated  by  greater  than  0.5  m. 
None  of  the  common  detection  were  separated  by  more  than  1.05  m.  Figure  4- 13(c) 
shows  the  location  error  for  the  magnetometer  versus  that  of  the  EMI  system.  It  is  evident 
from  the  figure  that  the  location  errors  are  highly  clustered  within  an  area  of  radius  0.5  m 
surrounding  the  baseline  item.  There  is  also  a  suggestion  that  large  location  errors  for  the 
EMI  system  tend  to  be  correlated  with  large  location  errors  for  the  magnetometer.  But  the 
corollary  is  not  true.  There  are  no  statistically  significant  easting  or  northing  location 
offsets  [see  Figure  4- 13(b)].  Both  the  mean  and  the  median  are  within  the  0.05-m  location 
error  estimate  for  the  RTK-DGPS. 

The  overall  performance  suggests  that  the  location  errors  exhibited  by  the  sensors 
are  small  and  tend  to  be  similar.  The  location  error  appears  to  be  caused  by  survey-to- 
survey  location  inaccuracies  and  noise  and  not  by  a  statistically  significant  intrinsic 
property  of  the  ordnance,  the  MTADS  system,  or  the  MTR. 

4.3.1.2  Detection  Depth  Offset 

The  depth  offset  is  the  difference  between  the  fitted  depth  of  each  detected 
baseline  item  (on  the  x-y  plane)  for  the  two  sensors,  that  is,  the  target  depth  estimated 
from  the  EMI  data  subtracted  from  that  estimated  from  the  magnetometer  data.  The  depth 
offset  can  be  positive  or  negative.  The  trend  of  the  sign  indicates  whether  one  sensor 
consistently  estimates  the  depth  of  the  baseline  item  to  be  greater  than  the  fitted  depth 
from  the  other  sensor. 

In  the  ME  sensor  set,  34  of  48  baseline  items  have  a  depth  offset  within  0.5  m  of 
one  another,  and  15  of  these  are  within  0.25  m.  There  are  seven  detections  with  depth 
offsets  between  1.0  and  2.0  m  [see  Figure  4-14(a)];  this  is  due  to  the  relatively  poor  depth 
predictions  from  the  EMI  sensor  data  (and  considering  the  excellent  location  accuracy, 
probably  due  to  an  ineffective  model  for  calculating  depth).  The  general  trend  is  that  the 
EMI  analysis  produces  a  depth  that  is  less  than  that  estimated  from  the  magnetometer 
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Figure  4-13.  Fitted  Location  Comparison  for  EMI  and  Magnetometer,  (a)  Distribution 
of  Distance  Between  EMI  and  Magnetometer  Location,  (b)  Difference  in  East 
versus  North,  (c)  Radial  Location  Error  for  Magnetometer  versus  EMI. 
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Figure  4-14.  Fitted  Depth  Comparison  for  EMI  and  Magnetometer,  (a)  Distribution  of 
Depth  for  EMI  and  Magnetometer,  (b)  Depth  Error  for  Magnetometer  versus  EMI. 
(c)  Radial  Distance  versus  Depth  Distance. 
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data.  For  the  10  ordnance  items  where  the  magnetometer  depth  estimation  is  less  than 
that  of  the  EMI,  only  3  common  detections  are  within  25  cm.  The  mean  depth  offset  for 
the  pair  is  -0.3 1  m  (the  negative  value  results  from  larger  depth  for  the  estimates  using 
the  magnetometer  data)  with  a  standard  deviation  of  0.46  m. 

Figure  4- 14(b)  shows  the  depth  error  for  the  EMI  sensor  data  relative  to  magne¬ 
tometer  sensor  data.  The  EMI  mean  depth  error  for  commonly  detected  targets  is  -0.25  m 
with  a  standard  deviation  of  0.44  cm,  while  the  magnetometer  has  a  mean  depth  error  of 
0.06  m  with  a  standard  deviation  of  0.35  m.  There  is  no  apparent  trend  between  large 
depth  error  in  one  sensor  and  large  depth  error  in  the  other. 

Figure  4- 14(c)  shows  the  radial  distance  between  the  two  detections  and  the  depth 
distance  between  detections.  There  is  no  apparent  correlation  between  large  differences  in 
depth  and  larger  radial  differences. 

4.3.1.3  Detection  GoF  Comparison 

Common  detections  for  both  the  magnetometer  and  EMI  exhibit  weakly  correla¬ 
ted  GoFs  [see  Figure  4-15(a)].  Only  1 1  common  detections  have  both  fits  >  90  percent 
(this  is  the  entire  set  of  EMI  fits  at  that  level),  while  25  common  detections  have  at  least 
one  GoF  parameter  >  90  percent.  The  conclusion  from  the  general  trends  is  that  the 
sources  of  poor  GoFs  are  dependent  on  the  type  of  sensor.  There  appears  to  be  little 
additional  information  available  to  strengthen  the  detection/discrimination  process  by 
comparison  of  GoFs. 


Figure  4-15.  Comparison  of  Fitted  Parameters  for  Magnetometer  and  EMI. 
(a)  Magnetometer  GoF  versus  EMI  GoF.  (b)  Ferrous  Size  for 
EMI  versus  Magnetometer  Ferrous  Size. 
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4.3.1.4  Detection  Size  Comparison 

The  ferrous  sizes  of  the  magnetometer  and  EMI  combination  are  correlated 
(correlation  coefficient  =  0.67)  [see  Figure  4-15(b)].  The  fitted  ferrous  size  of  all  but 
seven  magnetometer  detections  is  larger  than  the  corresponding  fitted  ferrous  size  for  the 
EMI  sensor. 

4.3.1.5  Targets  Detected  by  Either  Magnetometer  or  EMI 

Table  4-7  describes  ordnance  items  detected  by  one  sensor  only.  The  magne¬ 
tometer  tends  to  miss  shallow  emplaced  targets  (typically  small)  or  deeper  targets  at  or 
near  maximum  penetration,  while  the  EMI  tended  to  miss  ordnance  emplaced  at  greater 
depths  (these  can  be  large). 


Table  4-7.  Targets  Detected  by  Only  One  Sensor 


Ordnance 

Depth  (m) 

Missed  By 

8  in. 

2.32 

EMI 

60  mm 

0.38 

MAG 

60  mm 

0.85 

MAG 

105  mm 

0.30 

MAG 

250  lb 

2.10 

EMI 

250  lb 

2.16 

MAG 

250  lb 

2.53 

MAG 

500  lb 

3.66 

EMI 

500  lb 

4.27 

EMI 

4.3.2  False  Alarms  Common  to  the  ME  Pair 

To  determine  whether  there  are  trends  in  false-alarm  space,  we  analyze  the  false 
alarms  common  to  the  ME  sensor  pair.  In  this  case,  determining  conunon  false  alarms  is 
nontrivial  because  contrary  to  the  common  detection  sets,  the  source  and  location  of  the 
false  alarm  is  unknown.  Thus,  we  use  the  following  process  to  establish  the  common 
false  alarms.  The  full  declaration  set  for  each  sensor  is  parsed  into  two  subsets:  declara¬ 
tions  that  match  baseline  items  and  declarations  that  are  false  alarms.  The  two  false-alarm 
sets  are  then  compared  by  location  only  (in  the  x-y  plane).  A  false  alarm  from  one  set 
within  1.0  m  (the  critical  false-alarm  radius)  of  a  false  alarm  from  another  set  is  classified 
as  a  common  false  alarm  for  that  sensor  pair.  It  is  possible  to  have  more  than  one  false 
alarm  from  one  sensor  set  within  the  designated  critical  false-alarm  radius  of  the  second 
set,  but  only  the  nearest  neighbor  pair  is  considered  in  this  analysis.  We  choose  the  1.0-m 
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critical  false-alarm  radius  because  of  the  overwhelming  trend  to  common  detection  of  the 
baseline  items  of  less  than  0.5-m  location  offset;  1.0  m  is  more  than  three  standard  devia¬ 
tions  from  the  mean  separation  for  a  detection  pair. 

Using  this  set  of  common  false  alarms,  we  examine  the  sensor  pair  for  trends  in 
location  offset,  depth  offset,  GoF,  and  size.  The  trends  in  the  data  show  some  degree  of 
correlation  between  the  fitted  parameters,  which  confirms  our  hypothesis  that  the  vast 
majority  of  false  alarms  are  the  result  of  compact,  near-surface  anomalies  that  act  as 
clutter  for  the  sensor. 


4.3.2.1  False-Alarms  Location  Offset 


From  the  full  set  of  146  EMI  and  228  magnetometer  false  alarms,  there  are  47,  57, 
and  59  common  false  alarms  for  critical  false-alarm  radii  of  0.5,  0.75,  1.0  m,  respectively. 
Figure  4- 16(a)  shows  the  distribution  of  distance  between  the  false  alarm  locations  for  the 
two  sensors.  The  distribution  has  a  median  at  about  0.34  m.  There  is  the  potential  that 
false  alarms  that  are  not  really  common  could  be  identified  by  other  fit  parameters,  such 
as  depth.  Unluckily,  since  most  clutter  is  within  the  first  half  meter  of  the  ground  (see 
Figure  4-6),  false  alarms  from  two  different  sources  are  likely  to  have  similar  depths. 


Distance  between  False  Alarms  (m) 
(b) 


Difference  in  East  (m) 


Figure  4-16.  Fitted  Location  Comparison  for  EMI  and  Magnetometer  Common  False 
Alarms,  (a)  Distribution  of  Distance  Between  EMI  and  Magnetometer  Location, 
(b)  Difference  in  East  versus  North. 


4-22 


Figure  4- 16(b)  shows  the  location  offset  for  the  sensor  pair  in  the  east  versus 
north  directions.  As  with  the  ordnance,  the  mean  and  median  offsets  in  each  direction  are 
approximately  equal  to  the  5-cm  location  error  expected  from  the  RTK-DGPS.  In 
addition,  both  standard  deviations  are  approximately  30  cm. 

4.3.2.2  False-Alarm  Depth  Offset 

For  the  ME  sensor  combination,  the  distribution  of  depth  offsets  reveals  that  47  of 
the  59  common  false  alarms  have  a  depth  offset  between  ±0.25  m.  Figure  4-17  shows  the 
distribution  of  the  depth  offset.  The  EMI  sensor  appears  to  consistently  underestimate  the 
depth  of  the  target;  47  out  of  59  (80  percent)  common  false  alarms  have  the  EMI  detec¬ 
tion  depth  less  than  the  magnetic  detection  depth.  This  is  almost  identical  to  the  relative 
depth  for  the  commonly  detected  ordnance  (79  percent).  Thus,  there  is  no  apparent 
difference  between  the  depth  offset  for  ordnance  and  clutter. 


Distance  between  Depths  (m) 


Figure  4-17.  Distribution  of  Depth  Offset  for  EMi  and  Magnetometer 
Common  Faise  Aiarms 

Table  4-8  lists  the  12  ME  pairs  having  a  depth  offset  greater  than  ±0.25  m.  Except 
for  four  items,  the  fit  depths  of  each  sensor  are  within  0.5  m  of  the  surface.  Though  seven 
of  the  fit  parameters  for  the  magnetometer  false  alarms  are  greater  than  90,  the  respective 
fit  parameters  for  the  EMI  are  low.  The  last  common  false  alarm  listed  in  Table  4-8  with 
a  -3.17  m  depth  offset  is  probably  not  a  common  false  alarm  but  rather  the  detection  of 
two  closely  spaced  items  with  very  different  depths. 

4.3.2.3  False-Alarm  GoF  Comparison 

The  correlation  between  GoFs  of  the  ME  pairs  is  very  low,  perhaps  due  to  poor 
fits  from  the  EMI  data.  Only  two  common  false  alarms  had  a  GoF  greater  than  90  percent 
for  the  EMI,  and  both  of  these  had  GoFs  greater  than  90  percent  for  the  magnetometer. 
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Table  4-8.  GoF  for  Large  Depth  Offset 


Mag.  Fit 

Depth  (m) 

Mag. 

GoF 

EMI  Fit 

Depth  (m) 

EMI 

GoF 

Depth  Offset 
(m) 

0.10 

96 

0.62 

51 

0.52 

0.19 

93 

0.65 

53 

0.46 

0.11 

98 

0.45 

72 

0.34 

1.34 

89 

1.07 

83 

-0.20 

0.44 

61 

0.16 

76 

-0.27 

0.42 

81 

0.14 

43 

-0.28 

0.37 

93 

0.00 

43 

-0.37 

0.37 

93 

0.00 

55 

-0.37 

0.37 

97 

0.00 

16 

-0.37 

4.12 

96 

3.52 

16 

-0.60 

0.96 

87 

0.06 

72 

-0.90 

3.17 

79 

0.00 

69 

-3.17 

In  contrast  to  the  results  for  the  common  ordnance  detected,  many  declarations  with  a 
GoF  over  0.9  for  the  magnetometer  have  corresponding  EMI  GoF  of  less  than  0.50 
(17  out  of  43).  These  results  suggest  that  the  EMI  fitting  algorithm  has  a  harder  time 
fitting  clutter  objects,  which  are  likely  to  be  asymmetric.  At  the  same  time,  most  objects 
exhibit  a  strong  magnetic  dipole  moment  that  fits  well  to  the  simple  model  for  magnetic 
signatures.  Thus,  a  difference  in  GoFs  for  the  two  systems  might  provide  potential  for 
discrimination  using  the  “fused”  data  set.  It  should  be  noted  that  the  reciprocal  pattern  is 
not  true.  Good  GoF  for  the  EMI  always  corresponds  to  good  GoF  for  the  magnetometer. 

4.3.2.4  NRL  Ferrous  Size  Comparison 

The  average  fit  ferrous  size  for  the  magnetometer  is  8.7  cm,  compared  to  3.8  cm 
for  the  EMI.  Figure  4-18  shows  a  scatter  plot  of  EMI  ferrous  size  versus  magnetometer 
ferrous  size.  At  0.80,  the  correlation  is  quite  high,  but  the  EMI  consistently  under¬ 
estimates  size  relative  to  the  magnetometer.  For  56  out  of  the  59  common  false  alarms, 
the  ferrous  sizes  determined  from  the  magnetometer  data  were  greater  than  those 
determined  from  the  EMI. 

4.4  COMPARISON  OF  DETECTION  PERFORMANCE  FOR 
NRL  AND  IDA  DATA 

In  this  section  we  present  a  detection  performance  comparison  for  the  analysis 
product  generated  by  IDA  and  NRL  using  the  same  raw  sensor  data.  This  comparison 
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EMI  Ferrous  Size  (m) 

Figure  4-18.  Comparison  of  Fitted  Ferrous  Size  for  Magnetometer  versus  EMI  Ferrous  Size 

demonstrates  the  variability  between  users  of  the  same  analysis  software  package  (the 
MTADS-D AS).  Although  both  analysts  have  a  strong  background  in  the  phenomenology 
of  UXO  detection,  the  NRL  analyst  has  significantly  more  experience  over  many  test 
sites,  in  particular,  the  Twentynine  Palms  MTR.  The  result  is  a  slightly  better,  though  not 
statistically  significant,  overall  performance  of  NRL  with  the  magnetometer  and  EMI 
data  and  much  better  performance  with  the  magnetic  gradiometer  data. 

4.4.1  Detection  and  False-Alarm  Performance 

Figures  4-19(a)-(c)  show  the  relative  performance  of  the  IDA  and  NRL  target 
picking  using  the  ROC  formalism.  Table  4-9  presents  the  results  for  both  IDA  and  NRL 
for  an  of  1  m. 

NRL  consistently  outperforms  IDA  in  the  identification  of  ordnance.  The  largest 
difference  is  the  magnetometer.  Here,  even  with  the  implementation  of  the  magnetic 
moment  direction  discrimination,  the  NRL  detection  rate  is  superior.  On  the  other  hand, 
for  both  the  magnetic  gradiometer  and  EMI  system,  the  difference  in  performance  is  not 
as  great.  If  fact,  given  the  statistical  nature  of  the  detection  process,  the  differences  are 
not  significant. 
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Figure  4-19.  versus  Pfg  for  NRL  Processing  (Triangles)  and  IDA  Processing  (Squares). 
The  open  marks  are  for  the  full  declaration  set.  The  solid  symbols  result  from 
implementing  the  appropriate  discrimination  algorithms  for  each  data  set 
(as  described  in  this  chapter  and  Chapter  3)  to  a  confidence  level  of  2. 

(a)  Magnetometer,  (b)  Magnetic  Gradiometer.  (c)  EMI. 


Table  4-9.  Detection  and  False-Alarm  Performance  of  MTADS 
from  IDA  and  NRL  Analysis  with  a  1-m  Rem 


IDA 

NRL 

Sensor 

Pc 

FAR 

(m-^) 

P,a 

Pc 

FAR 

(m-2) 

Pfa 

Magnetometer 
(Full  Data  Set) 

0.726 

2.0  X  10-^ 

6.5x10-2 

0.823 

7.4x10-2 

2.4  X  10-2 

Magnetometer 
(Confidence  Level  2) 

0.677 

7.4x  10-® 

2.4  X  10-2 

0.790 

4.9  X  10-2 

1.5  X  10-2 

Magnetic  Gradiometer 
(Full  Data  Set) 

0.710 

8.3x  10-® 

2.6x10-2 

0.774 

7.7x10-2 

2.4  X  10-2 

Magnetic  Gradiometer 
(Confidence  Level  2) 

0.532 

2.6x  10-® 

8.1  X  10-2 

0.726 

4.9x10-2 

1.6x10-2 

EMI 

0.823 

9.5  X  10'^ 

2.0  X  10-2 

0.839 

4.8x10-2 

1.5  X  10-2 
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4.4.2  Comparison  of  Missed  Targets 

Ordnance  items  detected  by  IDA  but  not  by  NRL  (and  vice  versa)  are  refit  using 
the  MTADS-DAS  software  to  determine  their  general  characteristics.  The  results  follow. 

4.4.2. 1  Magnetometer 

A  total  of  44  ordnance  items  were  detected  by  both  IDA  and  NRL.  Table  4-10 
lists  targets  missed  by  one  of  the  groups  using  the  magnetometer  data.  IDA  missed  those 
ordnance  items  with  a  small  dipole  signature  or  in  a  cluttered  environment.  In  addition, 
two  large  ordnance  items,  a  500-lb  bomb  and  a  750-lb  bomb,  were  not  detected. 


Table  4-10.  Ordnance  Found  by  Only  One  Group  Using  the  Magnetometer  Data 


Ordnance 

Depth  (m) 

Missed  By 

Refit  Comment 

8  in. 

2.32 

IDA 

Weak  Anomaly 

8  in. 

1.83 

IDA 

Weak  Anomaly 

8  in. 

1.95 

IDA 

Weak  Anomaly 

60  mm 

0.46 

IDA 

Fit  outside  1  -m  R^rg 

105  mm 

0.55 

IDA 

Not  surveyed 
completely 

250  lb 

2.53 

NRL 

Weak  Anomaly 

500  lb 

4.27 

IDA 

Fit  outside  1  -m  Rcrg 

750  lb 

3.96 

IDA 

Weak  Anomaly 

4.4.2.2  Gradiometer 

In  the  case  of  the  gradiometer,  IDA  correctly  identified  three  targets,  though  the 
location  accuracy  of  the  fitting  algorithm  was  outside  the  1-m  halo.  In  addition,  one 
155-mm  item  was  not  detected  because  part  of  its  dipole  signature  extended  into  an 
unsurveyed  area  (see  Table  4-11). 

4.4.2.3  EMI 

As  expected,  the  EMI  system  had  difficulty  fitting  those  ordnance  items  emplaced 
at  greater  depths.  Table  4-12  lists  the  ordnance  detected  by  only  one  group.  All  three 
ordnance  items  were  deeper  than  2  m. 
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Table  4-11.  Ordnance  Found  by  Only  One  Group  Using  the  Magnetic  Gradiometer  Data 


Ordnance 

Depth  (m) 

Missed  By 

Refit  Comment 

8  in. 

1.83 

IDA 

Weak  Anomaly 

8  in. 

2.32 

NRL 

Faint  Signature 

60  mm 

0.30 

IDA 

Faint  signature 

155  mm 

1.04 

NRL 

Weak  Anomaly 

155  mm 

1.19 

IDA 

Near  unsurveyed  area 

250  lb 

2.38 

NRL 

Faint  signature 

500  lb 

3.66 

IDA 

Refit  outside  1-m  halo 

500  lb 

3.96 

IDA 

Refit  outside  1  -m  halo 

500  lb 

2.23 

IDA 

Refit  outside  1-m  halo 

750  lb 

3.96 

IDA 

Faint  signature 

Table  4-12.  Ordnance  Found  by  Only  One  Group  Using  the  EMI  Data 


Ordnance 

Depth  (m) 

Missed  By 

8  in. 

2.32 

IDA 

8  in. 

2.74 

NRL 

250  lb 

3.11 

IDA 

4.5  FALSE  ALARMS  UNIQUE  TO  EACH  SENSOR 

Exploiting  characteristics  of  the  false  alarms  unique  to  each  sensor  is  a  potential 
avenue  for  improved  overall  detection  performance.  For  the  EMI  and  magnetometer  data, 
we  parse  the  full  set  of  false  alarms  to  provide  a  list  of  false  alarms  not  within  the  1-m 
false-alarm  critical  radius  (see  Section  4.3.2).  Comparing  the  fitted  parameters  for  the 
unique  and  common  false  alarms,  we  conclude  that  there  are  no  compelling  trends  to 
differentiate  false  alarms  identified  by  both  the  magnetometer  and  the  EMI  from  those 
that  are  only  found  by  a  single  detector. 

Figures  4-20(a)  and  (b)  show  the  depth  of  the  false  alarms  for  the  magnetometer 
and  EMI  system,  respectively.  Here,  the  number  of  false  alarms  in  each  depth  bin  is 
normalized  to  the  total  number  of  false  alarms  for  the  sensor  (either  unique  or  common). 
Table  4-13  lists  the  mean  and  standard  deviation  for  the  depths.  For  the  magnetometer, 
the  mean  common  false-alarm  depth  is  slightly  less  than  that  of  the  unique  false  alarms. 
That  difference  is  a  result  of  the  insensitivity  of  the  EMI  system  to  deeper  objects. 
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0  0.5  1  1.5  2  2.5  3 


Depth  (m) 

Figure  4-20.  A  Comparison  of  the  Distribution  of  Depths  for  Unique  (Solid)  and 
Common  (Hashed)  False  Alarms,  (a)  Magnetometer,  (b)  EMI. 


Table  4-13.  Comparison  of  Fitted  Parameters  for  Common  and  Unique  False  Alarms 


Mean 

Depth 

(m) 

STD^ 

Depth 

(m) 

Mean  Size 
(cm) 

STD 

Size 

(cm) 

Mean  Rel. 
Inclination 
(deg) 

STD 

Inclination 

(deg) 

Unique 

0.43 

0.96 

8.1 

9.6 

31.8 

42.2 

Magnetometer 

Common 

0.31 

0.69 

8.6 

9.0 

47.8 

34.5 

Magnetometer 

Unique  EMI 

0.12 

0.54 

3.3 

4.7 

— 

— 

Common  EMI 

0.16 

0.49 

3.8 

5.1 

— 

— 

^  STD  is  standard  deviation. 


Figures  4-2 1(a)  and  (b)  show  the  estimated  ferrous  size  of  the  source  of  the 
signature  measured  by  the  sensors.  For  both  sensors,  the  average  size  of  the  common 
false  alarms  is  slightly  larger  than  that  of  the  unique  false  alarms,  but  given  the  variance 
of  the  size  for  these  sets,  the  difference  is  statistically  insignificant.  As  is  the  case  with 
ordnance,  the  size  estimated  from  the  EMI  data  is  smaller  than  that  estimated  from  the 
magnetometer  data. 
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Ferrous  Size  (cm) 


Ferrous  Size  (cm) 

Figure  4-21 .  A  Comparison  of  the  Distribution  of  Sizes  for  Unique  (Solid)  and 
Common  (Hashed)  False  Alarms,  (a)  Magnetometer,  (b)  EMI. 

The  relative  inclination,  which  is  used  in  Chapter  3  to  sort  ordnance  from  clutter, 
shows  a  slight  trend  towards  higher  values  for  common  false  alarms  relative  to  unique 
false  alarms  for  the  magnetometer.  Figure  4-22  shows  the  relative  inclination  for  the  two 
classes  of  false  alarms.  Table  4-13  lists  the  mean  and  the  standard  deviation  of  the 
relative  inclination.  It  should  be  noted  that  40  percent  of  the  false  alarms  unique  to  the 
magnetometer  have  a  relative  inclination  less  than  the  30-deg  cutoff  used  in  the  analysis 
presented  in  Section  4.1.1.  This  compares  to  only  18  percent  for  the  common  false 
alarms.  It  should  also  be  noted  that  none  of  the  unique  magnetometer  detections  had  a 
relative  inclination  below  39  deg.  The  difference  in  the  distribution  of  the  relative 
inclination  for  the  two  classes  of  false  alarms  suggests  that  this  feature  might  be 
employed  in  conjunction  with  simple  and/or  logical  data  features. 

4.6  SIMPLE  AND/OR  SENSOR  FUSION 

The  simplest  method  for  data  fusion  is  to  use  a  simple  logic  gate  (AND  or  OR). 
This  decision-based  fusion  can  lead  to  higher  detection  rates  (the  OR  process)  or  lower 
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Relative  Inclination  (degrees) 

Figure  4-22.  A  Comparison  of  the  Relative  inclination  for  the  Magnetometer 
for  Unique  (Solid)  and  Common  (Hashed)  Faise  Alarms 


false-alarm  rates  (the  AND  process).  It  is  valuable  to  assess  the  overall  performance  of 
the  MTADS  using  the  simple  decision-based  data  fusion.  Figure  4-23  shows  the 
probability  of  detection  and  the  probability  of  false  alarm  for  the  individual  sensors  and 
the  fused  data.  Table  4-14  lists  the  detection  performance  for  the  sensors  and  the  fusion 
methods.  Neither  the  AND  nor  the  OR  fusion  is  better  (within  the  statistical  error)  than 
the  best  sensor,  the  EMI. 


Figure  4-23.  versus  Pfg  for  the  Magnetometer  (Square),  EMI  (Diamond), 
Sensor  AND  (Circle),  and  Sensor  OR  (Triangle) 
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Table  4-14.  Detection  Performance  for  the  Sensors  and  Fusion  Methods 


P, 

P,s 

d 

Magnetometer 

0.823 

0.024 

2.90 

EMI 

0.839 

0.015 

3.16 

Sensor  AND 

0.758 

0.006 

3.21 

Sensor  OR 

0.903 

0.033 

3.14 

To  improve  performance,  feature-level  fusion  must  employed.  That  is,  the  two 
sets  of  features  from  sensors  should  have  a  measurable  degree  of  orthogonality.  As 
shown  in  Section  4.3.1,  the  magnetometer  and  the  EMI  have  a  slightly  different  list  of 
ordnance  detected  on  the  MTR.  The  EMI  has  a  very  robust  detection  rate  for  small, 
shallow  ordnance  (and  the  magnetometer  does  not);  the  magnetometer  has  better 
performance  for  large,  deep  ordnance.  Still,  this  type  of  orthogonality  is  not  enough, 
since  some  features  are  needed  for  false-alarm  mitigation.  In  addition,  highly  cluttered 
environments  will  lead  to  undetected  ordnance,  even  within  the  set  of  ordnance  best 
detected  by  the  sensor.  Table  4-15  lists  the  ordnance  missed  by  both  the  magnetometer 
and  the  EMI  at  the  MTR.  All  these  ordnance  items  were  deeply  emplaced  bombs 
obscured  in  some  manner  by  the  cluttered  environment. 


Table  4-15.  Ordnance  Missed  by  Magnetometer  and  EMI 


Ordnance  Type 

Depth  (m) 

2,000-lb  Bomb 

4.88 

1 ,000-lb  Bomb 

5.09 

500-lb  Bomb 

4.42 

500-lb  Bomb 

3.87 

250-lb  Bomb 

3.08 
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GLOSSARY 


DAQ 

data  acquisition  system 

DAS 

data  analysis  system 

EMI 

electromagnetic  induction 

ESTCP 

Environmental  Security  Technology  Certification  Program 

FAR 

false-alarm  rate 

GoF 

goodness  of  fit 

IDA 

Institute  for  Defense  Analyses 

MCAGCC 

Marine  Corps  Air  Ground  Combat  Center 

ME 

magnetometer  and  EMI  pair  of  sensors 

MTADS 

Multi-sensor  Towed  Array  Detection  System 

MTR 

Magnetic  Test  Range 

NRL 

Naval  Research  Laboratory 

ROC 

Receiver  Operating  Characteristic 

RTK-DGPS 

real-time  kinematic  differential  global  position  system 

UXO 

unexploded  ordnance 
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APPENDIX  A 


MODELED  MAGNETIC  SIGNATURES  OF 
EMPLACED  ORDNANCE 


The  magnetic  dipole  moment  of  each  ordnance  item  emplaced  and  used  as  part  of 
the  baseline  (see  Chapter  3)  at  the  Magnetic  Test  Range  has  been  modeled  assuming  a 
purely  induced  magnetic  signature.  Table  A-1  presents  the  modeled  magnetic  dipole 
moments  for  62  ordnance  items  emplaced,  as  well  as  the  modeled  maximum  magnetic 
signature.  For  this  calculation  the  ordnance  is  assumed  to  be  solid. 
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Table  A-1.  Modeled  Magnetic  Moments  for  Ordnance  at  the  MTR 
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